## 4 Machine Learning-Based Label Quality Assurance
### Uncertainty Regions for Determining Good and Bad Labels  
During the training process, the machine learning-based label quality assurance neural network is exposed to both positive and negative samples. Positive samples comprise a pair of the original image and a good label, while negative samples consist of the original image and a bad label. To comprehend the distinction between good and bad labels, we first introduce the concept of uncertainty regions. These regions represent the area surrounding ground truth labels, delineating the space where all acceptable labels for a given object can reside. Figure 8 illustrates the ground truth label for a given image. Given that \(x1\) and \(y1\) denote the top-left coordinates of the ground truth bounding box label and \(x2\) and \(y2\) represent the bottom-right coordinates, the uncertainty region for coordinate \(x1\) is defined as follows:  
\[[x1-width\times 0.2,x1+width\times 0.0], \tag{1}\]  
for \(y1\):  
\[[y1-height\times 0.2,y1+height\times 0.0], \tag{2}\]  
for \(x2\):  
\[[x2+width\times 0.2,x2-width\times 0.0], \tag{3}\]  
and for \(y2\):  
\[[y2+height\times 0.2,y2-height\times 0.0], \tag{4}\]  
where _width_ and _height_ are the width and height of the input image, and \(x1\), \(y1\), \(x2\) and \(y2\) are absolute coordinates; \(x1\) and \(x2\) are in the \([0,width]\) range, and \(y1\) and \(y2\) in the \([0,height]\) range. Figure 7: A labeled sample from MS COCO belonging to the training set used for building the label quality assurance model. Throughout the training process, positive and negative samples are randomly generated from a single pair of raw image and the ground truth label. A good label is fully enclosed within the uncertainty region, i.e., inside the outer (red) bounding box and outside the inner (green) bounding box, as illustrated in Figure 9. All bounding box coordinates must be located within their corresponding uncertainty region ranges. Figure 10 displays an example of a good label situated within the outer bounding box and outside the inner bounding box, as indicated in blue. Figure 8: Ground truth for the object “elephant” in the image. Figure 9: Uncertainty region for the object “elephant” in the image. In contrast, a bad label is any label that lies partially or entirely outside the uncertainty region: either outside the red bounding box or inside the green bounding box, as shown in Figure 11. At least one of the bounding box coordinates must fall outside its corresponding uncertainty region range. It is crucial to note that one uncertainty region corresponds to a single bounding box label. If multiple bounding boxes are labeled and are intended to be fully enclosed within a single uncertainty region, that sample is considered a negative sample containing a bad label. Furthermore, if a sample contains multiple labeled bounding boxes and at least one of them is a bad bounding box, it is deemed a negative sample. The MS COCO dataset provides annotated bounding boxes, with each bounding box belonging to one of 80 classes. Figure 11: Example of a bad bounding box label for object “elephant” outside the uncertainty region. \[[y1-height\times 0.2,y1+height\times 0.0], \tag{2}\]0 Example of a good bounding box label for object “elephant” inside the uncertainty region. Each uncertainty region is associated with a specific class. The space for bad bounding box labels is extensive. To manage this vastness, error types are introduced to systematically cover the most common errors annotators may make. Error types are divided into two major categories: Type A, which includes labels placed inside uncertainty regions that are damaged or manipulated; and Type B, which involves the creation of new labels outside of uncertainty regions. Table 1 displays the subtypes A1, A2, A3, B1 and B2 of the two major types: Type A (damage to labels inside uncertainty regions) and Type B (creation of new labels outside of uncertainty regions). Figure 12 presents examples of Type A errors, and Figure 13 displays examples of Type B errors.