# Conceptual Cognitive Modeling for Fine-Grained Annotation Quality Assessment of Object Detection Datasets

Authors: Lei Guo; Xinying Xu; Gang Xie; Jerry Gao
Journal: Discrete Dynamics in Nature and Society
 (2020)

---

## Body

## 1. Introduction

In supervised learning, annotation quality plays a vital role in training and assessment of the models for several computer vision tasks such as object classification [1, 2], detection [3–6], and segmentation [7–9]. The training of object detection models relies on accurate and sufficient annotations. For large-scale object detection datasets, annotations are usually obtained through crowdsourcing platforms, which results from anonymous participants, and can be collected for efficiency [10–12]. However, due mainly to the untrained participants involved in the professional and time-consuming annotation tasks, this has inevitably led to subjective inconsistency and relatively low quality of the collected annotations. As a result, the annotation quality cannot be guaranteed, where the quality assessment of such annotations becomes a challenge in this context.Annotation quality in object detection is a specialized-purpose data quality problem. Data quality has been widely studied since the 1980s [13]. According to [14], data quality can be defined as the degree to which a set of characteristics of data fulfills the requirements. Data with high quality should represent the real-world entities accurately in the structure and fit for their intended uses. Besides, data quality is of multidimensional characteristics. By reviewing the related literature [14–19], a core set of data quality dimensions is defined, including the completeness, accuracy, and consistency. Moreover, there are a fair number of researches about annotation quality. Regarding the annotation quality in classification, accuracy is employed generally [20], not considering the hierarchy of categories. For annotation quality in object detection, quality is evaluated by Intersection-over-Union (IoU) [21]. IoU is the ratio of the intersection area of the ground truth and human annotation to the total area, only considering the quality of the bounding box [22]. There are few systematic researches about annotation quality of object detection. Consequently, we refer to general-purpose data quality and construct an annotation quality framework.To date, there are relatively few works reported on this topic. This is only addressed from the perspectives of the object category and IoU [21]. However, a few general-purpose metrics can also be applied for annotation quality assessment. And we should perform annotation quality assessment from various aspects of the two attributes: bounding box and label.Evaluation measures for object classification, detection, and segmentation could serve as a reference for annotation quality in object detection. Regarding flat object classification, precision and recall are employed to assess the performance [23–26]. As for hierarchical object classification, distance in the tree or the directed acyclic graph (DAG) is used to assess the performance [27–30]. The distance can treat the prediction errors differently. In terms of object detection, the mAP is usually employed [31–36], integrating precision, recall, and IOU. The mAP is calculated according to the predicted results and confidence scores. However, for annotations, reasonable confidence scores are hard to obtain. As a result, in this paper, we employ the metrics of precision and recall. Regarding object segmentation, evaluation measures can be categorized into three types: area-based measures, location-based measures, and combined measures [37–41]. These image segmentation measures pay more attention to the details and the intrinsic visual characteristics. Consequently, the idea of image segmentation evaluation is introduced into the annotation quality assessment framework.In this paper, we propose a fine-grained framework for annotation quality assessment of object detection datasets, containing three dimensions: accuracy, completeness, and consistency. First, we construct the basic quality assessment framework based on the core general-purpose data quality (DQ) measurement, including accuracy and completeness, which considers the characteristics of annotation. For consistency, we find that it is difficult to give a strict definition. Further, the relationship of classes should be considered. Previous literature indicates that the cognition of humans is hierarchical in concept [42, 43] and consistent in space-time representations [44–46]. Inspired by these observations, the consistency of bounding box, completeness of category, hierarchical accuracy of label, and consistency of label are extracted as four additional elements for annotation quality assessment. The main contributions of this paper are as follows:(1)
We present a fine-grained annotation quality assessment (FGAQA) framework for evaluating the quality of object detection datasets. By analyzing the characteristics of the attributes of the bounding box and the corresponding label, the annotation quality contains three dimensions: accuracy, completeness, and consistency.(2)
To tackle the limitations of the basic quality assessment framework, we introduce the theory of cognitive perception to analyze the annotation quality and add four elements of annotation quality, including the consistency of bounding box, completeness of category, hierarchical accuracy of the label, and consistency of label. Specifically, the hierarchical accuracy of the label can treat annotation errors distinctively and softly.(3)
Comprehensive case studies on the Urban Traffic Surveillance (UTS) dataset and the PASCAL VOC 2007 detection dataset verify the effectiveness of the proposed annotation quality assessment framework. We find that the ground truth annotations of the UTS dataset have more quality issues, compared to the ones of the PASCAL VOC 2007 detection dataset.The rest of this paper is organized as follows. In Section2, the proposed cognitive-driven FGAQA framework is presented in detail. Section 3 discusses experiments as two case studies on the UTS and PASCAL VOC datasets. Finally, concluding remarks and future work are given in Section 4.


## 2. Annotation Quality Assessment Framework

A novel annotation quality assessment framework in object detection is given in this section, which is shown in Figure1. The annotation has two attributes: bounding box and label. Annotation quality depends on its characteristics. For the bounding box, the size, location, and quantity could have some quality issues. Regarding the label, there may exist the quality problems of value and quantity. And the annotation quality serves reference for the training of the object detection model. Therefore, we define the quality dimensions according to the quality problems and the use of annotation. Inspired by some existing work [14–19], the dimensions of completeness, accuracy, and consistency are selected as the core set of the data quality dimensions. By considering the theory of cognitive perception, we redefine some elements based on annotation characteristics. As a result, a fine-grained annotation quality assessment framework is proposed, as shown in Figure 1. The framework is constructed from the views of the bounding box and label. Regarding the quality of the bounding box, completeness, accuracy, and consistency are defined. The completeness of the bounding box can be divided into the completeness of the bounding box’s quantity and the completeness of the bounding box’s size. In terms of the quality of the label, we define completeness, accuracy, and consistency. The completeness of the label consists of the completeness of the bounding box’s label and the completeness of the category. The accuracy of the label contains flat and hierarchical accuracy. And most of these dimensions are computed for every object and are averaged for an image and the total dataset.Figure 1
Annotation quality evaluation framework.


### 2.1. Annotation Quality of Bounding Box’s Quantity

#### 2.1.1. Completeness of Bounding Box

The dimension can be defined as the extent to which bounding boxes are of sufficient quantity and coverage degree for the object. The dimension of completeness focuses on the null values. As for the completeness of the bounding box’s quantity, the null values correspond to unannotated objects. In an object detection dataset, small objects are often be neglected. During the modeling process of object detection, the unannotated objects would be regarded as background. For the completeness of the bounding box’s size, the null values correspond to the uncovered areas of the bounding boxes.(1)
Completeness of bounding box’s quantity: for imagei, completeness of bounding box’s quantity is a metric that can be defined as follows:(1)CBiQuantity=niHuni,whereni is the true object number and niHu is the number of human annotations, namely, the number of bounding boxes. For the dataset, CBQuantity is(2)CBQuantity=∑i=1,…,NNCBiQuantityN,whereN is the number of images in the dataset.(2)
Completeness of bounding box’s size: the completeness of the bounding box’s size is a pixel-count-based metric and can be defined as follows. For thejth object in image i, the metric is(3)CBijSize=SijIntSijObj,whereSijInt is the intersection area of the object and bounding box, and SijObj is the area of the object. For image i, CBiSize is(4)CBiSize=∑j=1,…,niHuniHuCBijSizeniHu.For the dataset,CBSize is(5)CBSize=∑i=1,…,NNCBiSizeN.


#### 2.1.2. Accuracy of Bounding Box

The dimension is intended to measure the closeness of the bounding box to the object. When the accuracy is low, the bounding box contains too much background affecting the distinction between the object and the background. For the bounding box ofjth object in image i, the accuracy is(6)AccBij=SijIntSijBB,where SijBB is the area of the bounding box. In image i, the accuracy is(7)AccBi=∑j=1,…,niHuniHuAccBijniHu.For a dataset, the accuracy can be given as follows:(8)AccB=∑i=1,…,NNAccBi∑i=1,…,NNniHu.


#### 2.1.3. Consistency of Bounding Box

The dimension focuses on the violation of spatiotemporal continuity of size and location. In crowdsourcing platforms, bounding boxes in adjacent frames may be drawn by different workers. As a result, they could conflict in size and location. Faced with the case, we can perform a quality assessment of the consistency of the bounding box during the corresponding postprocessing. Afterward, the annotations would satisfy the constraints. Concretely, for example, if an object moves toward the camera parallelly, the constraints are as follows:(9)xcenterprevious≈xcentercurrent≈xcenternext,ycenterprevious≤ycentercurrent≤ycenternext,wprevious≤wcurrent≤wnext,hprevious≤hcurrent≤hnext,where xcenter and ycenter are the coordinates for the center of the bounding box, and w and h are the width and height of the bounding box. When the jth object in image i satisfies the constraints, the metric ConBij = 1. Otherwise, ConBij = 0. For image i, the consistency is(10)ConBi=∑j=1,…,niHuniHuConBijniHu.For the dataset, ConB is(11)ConB=∑i=1,…,NNConBiN.


### 2.2. Annotation Quality of Label

#### 2.2.1. Completeness of Label

The dimension can be split into two types. The completeness of the bounding box’s label is employed to measure if each box has a label. The completeness of category describes the completeness for the category’s quantity from the aspect of computational learning theory. In the common benchmarks for object detection, there exist minority categories. For a category, if the metric does not meet the requirement, the detection accuracy would be affected.(1)
Completeness of bounding box’s label: for imagei, the completeness is(12)CLi=niLabelniHu,whereniLabel is the number of labels. For a dataset, the metric is(13)CL=∑i=1,…,NNCLiN.(2)
Completeness of category: the completeness of category is a metric that measures whether the number of samples can meet the training for the object detection model. As for a dataset, the classes are usually organized in a semantic hierarchy tree. Regarding a leaf node, if it meets the conditionnleaf>nlowbound, the completeness is 1. Otherwise, the completeness is 0. For a parent node, the completeness is(14)CCparentLabel=∑k=1,…,nchildnchildCCkLabelnchild,wherenchild is the number of the corresponding child nodes. As a result, we can have the completeness of the category for a dataset.


#### 2.2.2. Accuracy of Label

The dimension is employed to measure the closeness of the human and ground truth annotations. Regarding a dataset collected by a crowdsourcing annotation platform, the label noise is the most common error and has a direct influence on the training of the object detection model. The dimension has two elements: flat accuracy and hierarchical accuracy. The flat accuracy of the label is the usual element. However, the label space is often hierarchical. The hierarchical element can treat annotation errors distinctively and is the foundation of the utilization of annotation errors. As a result, we introduce these two kinds of elements for label accuracy evaluation.(1)
Flat accuracy of label: the flat accuracy of the label includes two metrics: precision and recall. The precision and recall of classt are(15)Pt=tpttpt+fpt,Rt=tptntGTr,wherentGTr is the number of ground truth annotations for class t, and tpt and fpt are the numbers of true positive objects and false-positive objects, respectively. For a dataset, precision can be calculated as follows:(16)P=∑t=1,…,MMPtM,which treats each class equally. And similarly, the recall is obtained.(2)
Hierarchical accuracy of label: the element also has two metrics. The metrics of classt are(17)HPt=∑k=1,…,niHuniHuansCk∩ansCk′/ansCk′1/pniHu,HRt=∑k=1,…,niGTrniGTransCk∩ansCk′/ansCk1/pniGTr,whereniHu and niGTr are the corresponding numbers of human and ground truth annotations, Ck and Ck′ denote the ground truth and human annotation labels, and ans(C) is the operation for computing ancestors for class C, p>0. Then, via macroaveraging the metrics for all classes, the hierarchical precision and recall can be calculated.


#### 2.2.3. Consistency of Label

Similar to the consistency of the bounding box, consistency of label concentrates on the confliction of spatiotemporal continuity of label. In the crowdsourcing platform, the labels in the adjacent frames often conflict due to the existence of low-level workers. If the label of an object is consonant with the labels in the previous and next frames, the metricConLobject is 1; otherwise, ConLobject is 0. For image i, the consistency is(18)ConLi=∑j=1,…,niLabelniLabelConLijniLabel.For the dataset,ConL is(19)ConL=∑i=1,…,NNConLiN.


## 2.1. Annotation Quality of Bounding Box’s Quantity

### 2.1.1. Completeness of Bounding Box

The dimension can be defined as the extent to which bounding boxes are of sufficient quantity and coverage degree for the object. The dimension of completeness focuses on the null values. As for the completeness of the bounding box’s quantity, the null values correspond to unannotated objects. In an object detection dataset, small objects are often be neglected. During the modeling process of object detection, the unannotated objects would be regarded as background. For the completeness of the bounding box’s size, the null values correspond to the uncovered areas of the bounding boxes.(1)
Completeness of bounding box’s quantity: for imagei, completeness of bounding box’s quantity is a metric that can be defined as follows:(1)CBiQuantity=niHuni,whereni is the true object number and niHu is the number of human annotations, namely, the number of bounding boxes. For the dataset, CBQuantity is(2)CBQuantity=∑i=1,…,NNCBiQuantityN,whereN is the number of images in the dataset.(2)
Completeness of bounding box’s size: the completeness of the bounding box’s size is a pixel-count-based metric and can be defined as follows. For thejth object in image i, the metric is(3)CBijSize=SijIntSijObj,whereSijInt is the intersection area of the object and bounding box, and SijObj is the area of the object. For image i, CBiSize is(4)CBiSize=∑j=1,…,niHuniHuCBijSizeniHu.For the dataset,CBSize is(5)CBSize=∑i=1,…,NNCBiSizeN.


### 2.1.2. Accuracy of Bounding Box

The dimension is intended to measure the closeness of the bounding box to the object. When the accuracy is low, the bounding box contains too much background affecting the distinction between the object and the background. For the bounding box ofjth object in image i, the accuracy is(6)AccBij=SijIntSijBB,where SijBB is the area of the bounding box. In image i, the accuracy is(7)AccBi=∑j=1,…,niHuniHuAccBijniHu.For a dataset, the accuracy can be given as follows:(8)AccB=∑i=1,…,NNAccBi∑i=1,…,NNniHu.


### 2.1.3. Consistency of Bounding Box

The dimension focuses on the violation of spatiotemporal continuity of size and location. In crowdsourcing platforms, bounding boxes in adjacent frames may be drawn by different workers. As a result, they could conflict in size and location. Faced with the case, we can perform a quality assessment of the consistency of the bounding box during the corresponding postprocessing. Afterward, the annotations would satisfy the constraints. Concretely, for example, if an object moves toward the camera parallelly, the constraints are as follows:(9)xcenterprevious≈xcentercurrent≈xcenternext,ycenterprevious≤ycentercurrent≤ycenternext,wprevious≤wcurrent≤wnext,hprevious≤hcurrent≤hnext,where xcenter and ycenter are the coordinates for the center of the bounding box, and w and h are the width and height of the bounding box. When the jth object in image i satisfies the constraints, the metric ConBij = 1. Otherwise, ConBij = 0. For image i, the consistency is(10)ConBi=∑j=1,…,niHuniHuConBijniHu.For the dataset, ConB is(11)ConB=∑i=1,…,NNConBiN.


## 2.1.1. Completeness of Bounding Box

The dimension can be defined as the extent to which bounding boxes are of sufficient quantity and coverage degree for the object. The dimension of completeness focuses on the null values. As for the completeness of the bounding box’s quantity, the null values correspond to unannotated objects. In an object detection dataset, small objects are often be neglected. During the modeling process of object detection, the unannotated objects would be regarded as background. For the completeness of the bounding box’s size, the null values correspond to the uncovered areas of the bounding boxes.(1)
Completeness of bounding box’s quantity: for imagei, completeness of bounding box’s quantity is a metric that can be defined as follows:(1)CBiQuantity=niHuni,whereni is the true object number and niHu is the number of human annotations, namely, the number of bounding boxes. For the dataset, CBQuantity is(2)CBQuantity=∑i=1,…,NNCBiQuantityN,whereN is the number of images in the dataset.(2)
Completeness of bounding box’s size: the completeness of the bounding box’s size is a pixel-count-based metric and can be defined as follows. For thejth object in image i, the metric is(3)CBijSize=SijIntSijObj,whereSijInt is the intersection area of the object and bounding box, and SijObj is the area of the object. For image i, CBiSize is(4)CBiSize=∑j=1,…,niHuniHuCBijSizeniHu.For the dataset,CBSize is(5)CBSize=∑i=1,…,NNCBiSizeN.


## 2.1.2. Accuracy of Bounding Box

The dimension is intended to measure the closeness of the bounding box to the object. When the accuracy is low, the bounding box contains too much background affecting the distinction between the object and the background. For the bounding box ofjth object in image i, the accuracy is(6)AccBij=SijIntSijBB,where SijBB is the area of the bounding box. In image i, the accuracy is(7)AccBi=∑j=1,…,niHuniHuAccBijniHu.For a dataset, the accuracy can be given as follows:(8)AccB=∑i=1,…,NNAccBi∑i=1,…,NNniHu.


## 2.1.3. Consistency of Bounding Box

The dimension focuses on the violation of spatiotemporal continuity of size and location. In crowdsourcing platforms, bounding boxes in adjacent frames may be drawn by different workers. As a result, they could conflict in size and location. Faced with the case, we can perform a quality assessment of the consistency of the bounding box during the corresponding postprocessing. Afterward, the annotations would satisfy the constraints. Concretely, for example, if an object moves toward the camera parallelly, the constraints are as follows:(9)xcenterprevious≈xcentercurrent≈xcenternext,ycenterprevious≤ycentercurrent≤ycenternext,wprevious≤wcurrent≤wnext,hprevious≤hcurrent≤hnext,where xcenter and ycenter are the coordinates for the center of the bounding box, and w and h are the width and height of the bounding box. When the jth object in image i satisfies the constraints, the metric ConBij = 1. Otherwise, ConBij = 0. For image i, the consistency is(10)ConBi=∑j=1,…,niHuniHuConBijniHu.For the dataset, ConB is(11)ConB=∑i=1,…,NNConBiN.


## 2.2. Annotation Quality of Label

### 2.2.1. Completeness of Label

The dimension can be split into two types. The completeness of the bounding box’s label is employed to measure if each box has a label. The completeness of category describes the completeness for the category’s quantity from the aspect of computational learning theory. In the common benchmarks for object detection, there exist minority categories. For a category, if the metric does not meet the requirement, the detection accuracy would be affected.(1)
Completeness of bounding box’s label: for imagei, the completeness is(12)CLi=niLabelniHu,whereniLabel is the number of labels. For a dataset, the metric is(13)CL=∑i=1,…,NNCLiN.(2)
Completeness of category: the completeness of category is a metric that measures whether the number of samples can meet the training for the object detection model. As for a dataset, the classes are usually organized in a semantic hierarchy tree. Regarding a leaf node, if it meets the conditionnleaf>nlowbound, the completeness is 1. Otherwise, the completeness is 0. For a parent node, the completeness is(14)CCparentLabel=∑k=1,…,nchildnchildCCkLabelnchild,wherenchild is the number of the corresponding child nodes. As a result, we can have the completeness of the category for a dataset.


### 2.2.2. Accuracy of Label

The dimension is employed to measure the closeness of the human and ground truth annotations. Regarding a dataset collected by a crowdsourcing annotation platform, the label noise is the most common error and has a direct influence on the training of the object detection model. The dimension has two elements: flat accuracy and hierarchical accuracy. The flat accuracy of the label is the usual element. However, the label space is often hierarchical. The hierarchical element can treat annotation errors distinctively and is the foundation of the utilization of annotation errors. As a result, we introduce these two kinds of elements for label accuracy evaluation.(1)
Flat accuracy of label: the flat accuracy of the label includes two metrics: precision and recall. The precision and recall of classt are(15)Pt=tpttpt+fpt,Rt=tptntGTr,wherentGTr is the number of ground truth annotations for class t, and tpt and fpt are the numbers of true positive objects and false-positive objects, respectively. For a dataset, precision can be calculated as follows:(16)P=∑t=1,…,MMPtM,which treats each class equally. And similarly, the recall is obtained.(2)
Hierarchical accuracy of label: the element also has two metrics. The metrics of classt are(17)HPt=∑k=1,…,niHuniHuansCk∩ansCk′/ansCk′1/pniHu,HRt=∑k=1,…,niGTrniGTransCk∩ansCk′/ansCk1/pniGTr,whereniHu and niGTr are the corresponding numbers of human and ground truth annotations, Ck and Ck′ denote the ground truth and human annotation labels, and ans(C) is the operation for computing ancestors for class C, p>0. Then, via macroaveraging the metrics for all classes, the hierarchical precision and recall can be calculated.


### 2.2.3. Consistency of Label

Similar to the consistency of the bounding box, consistency of label concentrates on the confliction of spatiotemporal continuity of label. In the crowdsourcing platform, the labels in the adjacent frames often conflict due to the existence of low-level workers. If the label of an object is consonant with the labels in the previous and next frames, the metricConLobject is 1; otherwise, ConLobject is 0. For image i, the consistency is(18)ConLi=∑j=1,…,niLabelniLabelConLijniLabel.For the dataset,ConL is(19)ConL=∑i=1,…,NNConLiN.


## 2.2.1. Completeness of Label

The dimension can be split into two types. The completeness of the bounding box’s label is employed to measure if each box has a label. The completeness of category describes the completeness for the category’s quantity from the aspect of computational learning theory. In the common benchmarks for object detection, there exist minority categories. For a category, if the metric does not meet the requirement, the detection accuracy would be affected.(1)
Completeness of bounding box’s label: for imagei, the completeness is(12)CLi=niLabelniHu,whereniLabel is the number of labels. For a dataset, the metric is(13)CL=∑i=1,…,NNCLiN.(2)
Completeness of category: the completeness of category is a metric that measures whether the number of samples can meet the training for the object detection model. As for a dataset, the classes are usually organized in a semantic hierarchy tree. Regarding a leaf node, if it meets the conditionnleaf>nlowbound, the completeness is 1. Otherwise, the completeness is 0. For a parent node, the completeness is(14)CCparentLabel=∑k=1,…,nchildnchildCCkLabelnchild,wherenchild is the number of the corresponding child nodes. As a result, we can have the completeness of the category for a dataset.


## 2.2.2. Accuracy of Label

The dimension is employed to measure the closeness of the human and ground truth annotations. Regarding a dataset collected by a crowdsourcing annotation platform, the label noise is the most common error and has a direct influence on the training of the object detection model. The dimension has two elements: flat accuracy and hierarchical accuracy. The flat accuracy of the label is the usual element. However, the label space is often hierarchical. The hierarchical element can treat annotation errors distinctively and is the foundation of the utilization of annotation errors. As a result, we introduce these two kinds of elements for label accuracy evaluation.(1)
Flat accuracy of label: the flat accuracy of the label includes two metrics: precision and recall. The precision and recall of classt are(15)Pt=tpttpt+fpt,Rt=tptntGTr,wherentGTr is the number of ground truth annotations for class t, and tpt and fpt are the numbers of true positive objects and false-positive objects, respectively. For a dataset, precision can be calculated as follows:(16)P=∑t=1,…,MMPtM,which treats each class equally. And similarly, the recall is obtained.(2)
Hierarchical accuracy of label: the element also has two metrics. The metrics of classt are(17)HPt=∑k=1,…,niHuniHuansCk∩ansCk′/ansCk′1/pniHu,HRt=∑k=1,…,niGTrniGTransCk∩ansCk′/ansCk1/pniGTr,whereniHu and niGTr are the corresponding numbers of human and ground truth annotations, Ck and Ck′ denote the ground truth and human annotation labels, and ans(C) is the operation for computing ancestors for class C, p>0. Then, via macroaveraging the metrics for all classes, the hierarchical precision and recall can be calculated.


## 2.2.3. Consistency of Label

Similar to the consistency of the bounding box, consistency of label concentrates on the confliction of spatiotemporal continuity of label. In the crowdsourcing platform, the labels in the adjacent frames often conflict due to the existence of low-level workers. If the label of an object is consonant with the labels in the previous and next frames, the metricConLobject is 1; otherwise, ConLobject is 0. For image i, the consistency is(18)ConLi=∑j=1,…,niLabelniLabelConLijniLabel.For the dataset,ConL is(19)ConL=∑i=1,…,NNConLiN.


## 3. Case Study

To verify the effectiveness of the quality framework, two case studies are conducted based on the UTS dataset [47] and PASCAL VOC 2007 detection dataset [48]. UTS dataset is a video dataset with varying illumination conditions and viewpoints. PASCAL VOC 2007 dataset is an image dataset and contains twenty categories. Note that a few dimensions of the quality assessment framework are not fit for the dataset. To acquire the annotations, we let a group of students fulfill the annotation work. Generally, ground truth annotations are employed as golden standard annotations. However, in the evaluation process, we find that, to a certain extent, the ground truth annotations have quality problems, especially for the UTS dataset. Consequently, ground truth annotations are evaluated, where human annotations are regarded as “ground truth annotations.” Additionally, to verify the completeness of category, the relationship between this metric and detection performance is studied by conducting object detection experiments.


### 3.1. Case Study for UTS Dataset

In this case study, the UTS dataset is utilized for verification. To reduce the amount of annotation labor, four shots are selected, and we annotate an image for every four or five images. Finally, the numbers of images in the four shots are 75, 120, 100, and 120 with 1166, 686, 639, and 919 objects, respectively. The evaluation is presented from the aspects of an image and a dataset. We find that the ground truth annotations have quality problems, especially for the completeness of the bounding box’s quantity and the flat recall of the label.


#### 3.1.1. Annotation Quality of an Image

For the clarity of the description of annotation quality, an image is selected for evaluation, which is given in Figure2. The semantic hierarchy tree we defined is presented in Figure 3. The quality evaluation results for an image are given in Table 1. The accuracy of the bounding box for each object is shown in Figure 4.Figure 2
Human and ground truth annotations from the UTS dataset (ground truth and human annotations are shown in red and yellow, respectively).Figure 3
Semantic hierarchy tree and completeness of category of ground truth annotations for the original UTS training dataset.Table 1
Results of other quality dimensions for an image.
Annotation quality dimension
Value
Completeness of bounding box’s quantity
1
Completeness of bounding box’s size
0.64
Accuracy of bounding box
0.67
Consistency of bounding box
1
Completeness of bounding box’s label
1
Flat precision/recall of label
–/0.5
Hierarchical precision/recall of label
0.69/0.79
Flat precision/recall of hatchback
0.25/1
Hierarchical precision/recall of hatchback
0.42/0.83
Consistency of label
0.84Figure 4
Accuracy of the bounding box in an image (note that two small objects are missed by the image instance segmentation algorithm.).Now, the analysis is given below. According to Table1, the flat precision of hatchback is 0.25. However, it is because of the quality problems of ground truth annotations. Reviewing the annotations, we find that there are two small unannotated objects as shown in Figure 2. Hierarchical measures can reflect the relation of the classes. For instance, hierarchical precision for the hatchback is 0.42, while the flat precision is 0.25. Further, the consistency of the label is less than 1. It shows that there are inconsistent labels with the labels in adjacent frames. In Table 1, four metrics are equal to 1, reflecting that there is no error from these aspects.


#### 3.1.2. Annotation Quality of Human and Ground Truth Annotations

Afterward, we show the annotation quality of the UTS dataset for the human and ground truth annotations. The annotation accuracies of the label are given in Tables2 and 3. The completeness of the category of the ground truth annotations for each class and the original vehicle dataset is given in Figure 3, where the threshold is set to 1000. The results of other quality dimensions are presented in Table 4.Table 2
Annotation accuracy of human annotations for the downsampled UTS dataset.
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Hatchback
0.79 ± 0.02
0.55 ± 0.14
0.92 ± 0.01
0.81 ± 0.07
Sedan
0.58 ± 0.08
0.78 ± 0.12
0.86 ± 0.03
0.88 ± 0.07
Minibus
0.93 ± 0.10
0.57 ± 0.33
0.95 ± 0.06
0.70 ± 0.22
SUV
0.20 ± 0.06
0.27 ± 0.10
0.69 ± 0.03
0.74 ± 0.04
MPV
0.19 ± 0.14
0.26 ± 0.14
0.62 ± 0.14
0.72 ± 0.08
Pickup
0.57 ± 0.41
1 ± 0
0.72 ± 0.27
1 ± 0
On average
0.55 ± 0.11
0.57 ± 0.07
0.79 ± 0.07
0.81 ± 0.05Table 3
Annotation accuracy of ground truth annotations for the downsampled UTS dataset.
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Hatchback
0.57 ± 0.13
0.57 ± 0.04
0.85 ± 0.06
0.67 ± 0.05
Sedan
0.79 ± 0.12
0.44 ± 0.04
0.91 ± 0.08
0.65 ± 0.02
Minibus
0.58 ± 0.34
0.77 ± 0.17
0.72 ± 0.23
0.79 ± 0.17
SUV
0.27 ± 0.10
0.15 ± 0.05
0.75 ± 0.04
0.53 ± 0.09
MPV
0.26 ± 0.14
0.17 ± 0.15
0.72 ± 0.08
0.51 ± 0.18
Pickup
1 ± 0
0.28 ± 0.19
1 ± 0
0.28 ± 0.19
On average
0.58 ± 0.07
0.40 ± 0.07
0.83 ± 0.04
0.57 ± 0.07Table 4
Results of other quality dimensions for the downsampled UTS dataset.
Annotation quality dimension
Human annotations
Ground truth annotations
Completeness of bounding box’s quantity
0.98 ± 0.02
0.75 ± 0.05
Completeness of bounding box’s size
0.96 ± 0.02
0.99
Consistency of bounding box
0.977 ± 0.002
0.96
Completeness of bounding box’s label
0.52 ± 0.11
0.58
Consistency of label
0.86 ± 0.01
0.71The quality of human annotations is analyzed first. According to Tables2 and 4, the overall annotation quality of the bounding box is good, while the annotation quality of the label is relatively poor. Accordingly, it can be inferred that the label’s annotation is a more difficult task. In particular, for SUV and MPV, the accuracy and recall are too low. The hierarchical accuracy is higher than the flat accuracy, treating errors distinctively. According to Table 4, compared with other dimensions, the consistency of the label is lower on account of the own property.The quality of ground truth annotations is evaluated here. According to Tables2–4, the completeness of bounding box’s quantity, flat and hierarchical recall of label, and consistency of label for ground truth annotations are lower than those for human annotations. When reviewing ground truth annotations, we find that ground truth annotations neglect some small and incomplete objects. But these small and incomplete objects can be annotated properly by experience. There are more inconsistent labels in ground truth annotations than in human annotations. Figure 3 shows that the completeness of category for MPV and pickup is 0, as the corresponding category’s quantities do not reach the threshold. Generally, the quality problem exists in the ground truth annotations. Therefore, it is significant to perform a quality assessment in the process of annotation and ground truth inference.


#### 3.1.3. Relationship between the Completeness of Category and Detection Performance

For the sake of exploring the relationship between the completeness of category and detection performance, the following experiment is conducted, which implies the effectiveness of the dimension. The object detection experiment on the UTS dataset is performed on the original dataset and downsampled dataset. As for downsampling, we just select images for every two images. The detection algorithm we use is Faster RCNN [3]. Table 5 presents the corresponding result.Table 5
Comparison of detection results based on the original training dataset and downsampled dataset.
Class
Object number in the training dataset
mAP (original)
mAP (downsampled)
Hatchback
12165
0.669
0.744
Sedan
5484
0.573
0.565
Minibus
3220
0.663
0.601
SUV
1761
0.560
0.576
MPV
898
0.154
0.142
Pickup
263
0.020
0.0001
On average
3965.2
0.440
0.438According to Table5, we argue that the detection result is closely related to the completeness of category. Overall, for the complete class whose training samples’ quantity is over 1000, the corresponding mAP is high, while the detection mAPs of other classes are quite low. However, for SUV in the downsampled dataset, the quantity is about 880. The detection performance is still acceptable. It is due to its salient visual feature. Thus, the threshold varies with the class. Additionally, for the incomplete class, the performance declines with downsampling.


### 3.2. Case Study for PASCAL VOC 2007 Detection Dataset

In the case study, PASCAL VOC 2007 detection dataset is utilized for verification. To save labor, we select twenty images for each class as annotation samples. Finally, a random-selected dataset containing 353 images is obtained. The PASCAL VOC 2007 dataset is an image dataset. Consequently, a few quality dimensions are not fit for the dataset.


#### 3.2.1. Annotation Quality for Human and Ground Truth Annotation

The quality of human and ground truth annotations for the PASCAL VOC 2007 dataset is given below. Accuracies of the label for the human and ground truth annotations are given in Tables6 and 7. The semantic hierarchy tree and completeness of category quantity are given in Figure 5, where the threshold is set as 400. The results of other quality dimensions are provided in Table 8.Table 6
Annotation accuracy of human annotations for the selected images of the PASCAL 2007 dataset (the average values are computed for the twenty classes).
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Person
0.98 ± 0.01
0.92 ± 0.05
0.99 ± 0.01
0.92 ± 0.04
Car
0.99 ± 0.02
0.94 ± 0.04
0.99 ± 0.01
0.94 ± 0.03
Chair
0.96 ± 0.03
0.74 ± 0.08
0.98 ± 0.01
0.82 ± 0.07
Bottle
0.98 ± 0.01
0.81 ± 0.08
0.99 ± 0.01
0.82 ± 0.07
Potted plant
1 ± 0
0.54 ± 0.31
1 ± 0
0.57 ± 0.29
Cow
0.99 ± 0.01
0.95 ± 0.01
0.997 ± 0.005
0.96 ± 0.01
Dining table
0.75 ± 0.16
0.59 ± 0.18
0.91 ± 0.06
0.64 ± 0.15
Bus
1 ± 0
0.94 ± 0.04
1 ± 0
0.96 ± 0.03
On average
0.96 ± 0.01
0.89 ± 0.04
0.983 ± 0.005
0.90 ± 0.04Table 7
Annotation accuracy of ground truth annotations for the selected images of the PASCAL 2007 dataset (the average values are computed for the twenty classes).
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Person
0.97 ± 0.01
0.79 ± 0.11
0.98 ± 0.01
0.81 ± 0.1
Car
0.94 ± 0.02
0.82 ± 0.08
0.96 ± 0.01
0.83 ± 0.08
Chair
0.90 ± 0.02
0.81 ± 0.09
0.96 ± 0.01
0.84 ± 0.08
Bottle
1 ± 0
0.74 ± 0.10
1 ± 0
0.81 ± 0.09
Potted plant
0.98 ± 0.02
0.77 ± 0.02
0.99 ± 0.01
0.78 ± 0.03
Cow
0.99 ± 0.01
0.81 ± 0.10
1 ± 0
0.83 ± 0.10
Dining table
0.68 ± 0.16
0.63 ± 0.11
0.87 ± 0.06
0.65 ± 0.09
Bus
0.91 ± 0.09
0.89 ± 0.08
0.94 ± 0.05
0.90 ± 0.07
On average
0.94 ± 0.02
0.84 ± 0.05
0.97 ± 0.01
0.87 ± 0.05Figure 5
Semantic hierarchy tree and completeness of category for original PASCAL VOC 2007 training dataset.Table 8
Results of other quality dimensions for the selected images and its original training dataset of the PASCAL VOC 2007 dataset.
Annotation quality dimension
Human annotations
Ground truth annotations
Completeness of bounding box’s quantity
0.90 ± 0.04
0.88 ± 0.05
Completeness of bounding box’s size
0.84 ± 0.02
0.85
Completeness of bounding box’s label
0.9991 ± 0.0008
1According to Tables6 and 8, we can see that the human annotation quality for the dataset is good overall. However, the accuracies of the chair, potted plant, and dining table are relatively poor. For instance, the average flat recall for the potted plant is 0.54. This is because the potted plant is small and tends to be neglected. And for the other dimensions of human annotations, quality is relatively reliable.Afterward, we evaluate the annotation quality of ground truth annotations. According to Tables6–8, we find that the quality of ground truth annotations is slightly worse than that of human annotations. Specifically, the completeness of the bounding box’s quantity and the flat recall of the label are relatively low. These dimensions indicate that there are more unannotated objects. As there are not enough images in the random-selected dataset, we calculate the completeness of category according to the original training set. The total completeness of category is 0.62, as 38% of the classes do not have enough samples.


#### 3.2.2. Relationship between the Completeness of Category and Detection Performance

To explore the relationship between the completeness of category and detection performance, an experiment is conducted in the same way as the previous section. We conduct object detection experiments on the original dataset and downsampled dataset of which the sampling ratio is 0.5. And the major classes of person, car, and chair are not downsampled. Table9 presents the detection results, where classes are in descending order of quantity of training samples.Table 9
Comparison of detection results based on the original training dataset and downsampled dataset (the average values are computed for the twenty classes).
Class
Object number in the training dataset
mAP (original)
mAP (downsampled)
Person
5447
0.779
0.778
Car
1644
0.831
0.807
Chair
1432
0.520
0.511
Bottle
634
0.576
0.519
Potted plant
625
0.459
0.376
Cow
356
0.767
0.721
Dining table
310
0.682
0.671
Bus
272
0.772
0.776
On average
783.1
0.714
0.678According to Table9, on the whole, the detection performance declines after the dataset is downsampled. For the majority classes of person, car, and chair, there are no obvious declines of mAPs, as we do not make downsampling on these classes. As for the minority classes, mAPs for the bottle and potted plant decline a lot, which can be regarded as hard classes. But mAPs for the other classes of the minority are relatively high and change little, which should be regarded as easy classes. The hard classes are usually of small scale and have nonsalient visual features, hindering the learning of the object detection model. Therefore, the threshold for hard classes is relatively high. In the future process of constructing a dataset, the training samples’ quantity for hard classes should be added.


## 3.1. Case Study for UTS Dataset

In this case study, the UTS dataset is utilized for verification. To reduce the amount of annotation labor, four shots are selected, and we annotate an image for every four or five images. Finally, the numbers of images in the four shots are 75, 120, 100, and 120 with 1166, 686, 639, and 919 objects, respectively. The evaluation is presented from the aspects of an image and a dataset. We find that the ground truth annotations have quality problems, especially for the completeness of the bounding box’s quantity and the flat recall of the label.


### 3.1.1. Annotation Quality of an Image

For the clarity of the description of annotation quality, an image is selected for evaluation, which is given in Figure2. The semantic hierarchy tree we defined is presented in Figure 3. The quality evaluation results for an image are given in Table 1. The accuracy of the bounding box for each object is shown in Figure 4.Figure 2
Human and ground truth annotations from the UTS dataset (ground truth and human annotations are shown in red and yellow, respectively).Figure 3
Semantic hierarchy tree and completeness of category of ground truth annotations for the original UTS training dataset.Table 1
Results of other quality dimensions for an image.
Annotation quality dimension
Value
Completeness of bounding box’s quantity
1
Completeness of bounding box’s size
0.64
Accuracy of bounding box
0.67
Consistency of bounding box
1
Completeness of bounding box’s label
1
Flat precision/recall of label
–/0.5
Hierarchical precision/recall of label
0.69/0.79
Flat precision/recall of hatchback
0.25/1
Hierarchical precision/recall of hatchback
0.42/0.83
Consistency of label
0.84Figure 4
Accuracy of the bounding box in an image (note that two small objects are missed by the image instance segmentation algorithm.).Now, the analysis is given below. According to Table1, the flat precision of hatchback is 0.25. However, it is because of the quality problems of ground truth annotations. Reviewing the annotations, we find that there are two small unannotated objects as shown in Figure 2. Hierarchical measures can reflect the relation of the classes. For instance, hierarchical precision for the hatchback is 0.42, while the flat precision is 0.25. Further, the consistency of the label is less than 1. It shows that there are inconsistent labels with the labels in adjacent frames. In Table 1, four metrics are equal to 1, reflecting that there is no error from these aspects.


### 3.1.2. Annotation Quality of Human and Ground Truth Annotations

Afterward, we show the annotation quality of the UTS dataset for the human and ground truth annotations. The annotation accuracies of the label are given in Tables2 and 3. The completeness of the category of the ground truth annotations for each class and the original vehicle dataset is given in Figure 3, where the threshold is set to 1000. The results of other quality dimensions are presented in Table 4.Table 2
Annotation accuracy of human annotations for the downsampled UTS dataset.
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Hatchback
0.79 ± 0.02
0.55 ± 0.14
0.92 ± 0.01
0.81 ± 0.07
Sedan
0.58 ± 0.08
0.78 ± 0.12
0.86 ± 0.03
0.88 ± 0.07
Minibus
0.93 ± 0.10
0.57 ± 0.33
0.95 ± 0.06
0.70 ± 0.22
SUV
0.20 ± 0.06
0.27 ± 0.10
0.69 ± 0.03
0.74 ± 0.04
MPV
0.19 ± 0.14
0.26 ± 0.14
0.62 ± 0.14
0.72 ± 0.08
Pickup
0.57 ± 0.41
1 ± 0
0.72 ± 0.27
1 ± 0
On average
0.55 ± 0.11
0.57 ± 0.07
0.79 ± 0.07
0.81 ± 0.05Table 3
Annotation accuracy of ground truth annotations for the downsampled UTS dataset.
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Hatchback
0.57 ± 0.13
0.57 ± 0.04
0.85 ± 0.06
0.67 ± 0.05
Sedan
0.79 ± 0.12
0.44 ± 0.04
0.91 ± 0.08
0.65 ± 0.02
Minibus
0.58 ± 0.34
0.77 ± 0.17
0.72 ± 0.23
0.79 ± 0.17
SUV
0.27 ± 0.10
0.15 ± 0.05
0.75 ± 0.04
0.53 ± 0.09
MPV
0.26 ± 0.14
0.17 ± 0.15
0.72 ± 0.08
0.51 ± 0.18
Pickup
1 ± 0
0.28 ± 0.19
1 ± 0
0.28 ± 0.19
On average
0.58 ± 0.07
0.40 ± 0.07
0.83 ± 0.04
0.57 ± 0.07Table 4
Results of other quality dimensions for the downsampled UTS dataset.
Annotation quality dimension
Human annotations
Ground truth annotations
Completeness of bounding box’s quantity
0.98 ± 0.02
0.75 ± 0.05
Completeness of bounding box’s size
0.96 ± 0.02
0.99
Consistency of bounding box
0.977 ± 0.002
0.96
Completeness of bounding box’s label
0.52 ± 0.11
0.58
Consistency of label
0.86 ± 0.01
0.71The quality of human annotations is analyzed first. According to Tables2 and 4, the overall annotation quality of the bounding box is good, while the annotation quality of the label is relatively poor. Accordingly, it can be inferred that the label’s annotation is a more difficult task. In particular, for SUV and MPV, the accuracy and recall are too low. The hierarchical accuracy is higher than the flat accuracy, treating errors distinctively. According to Table 4, compared with other dimensions, the consistency of the label is lower on account of the own property.The quality of ground truth annotations is evaluated here. According to Tables2–4, the completeness of bounding box’s quantity, flat and hierarchical recall of label, and consistency of label for ground truth annotations are lower than those for human annotations. When reviewing ground truth annotations, we find that ground truth annotations neglect some small and incomplete objects. But these small and incomplete objects can be annotated properly by experience. There are more inconsistent labels in ground truth annotations than in human annotations. Figure 3 shows that the completeness of category for MPV and pickup is 0, as the corresponding category’s quantities do not reach the threshold. Generally, the quality problem exists in the ground truth annotations. Therefore, it is significant to perform a quality assessment in the process of annotation and ground truth inference.


### 3.1.3. Relationship between the Completeness of Category and Detection Performance

For the sake of exploring the relationship between the completeness of category and detection performance, the following experiment is conducted, which implies the effectiveness of the dimension. The object detection experiment on the UTS dataset is performed on the original dataset and downsampled dataset. As for downsampling, we just select images for every two images. The detection algorithm we use is Faster RCNN [3]. Table 5 presents the corresponding result.Table 5
Comparison of detection results based on the original training dataset and downsampled dataset.
Class
Object number in the training dataset
mAP (original)
mAP (downsampled)
Hatchback
12165
0.669
0.744
Sedan
5484
0.573
0.565
Minibus
3220
0.663
0.601
SUV
1761
0.560
0.576
MPV
898
0.154
0.142
Pickup
263
0.020
0.0001
On average
3965.2
0.440
0.438According to Table5, we argue that the detection result is closely related to the completeness of category. Overall, for the complete class whose training samples’ quantity is over 1000, the corresponding mAP is high, while the detection mAPs of other classes are quite low. However, for SUV in the downsampled dataset, the quantity is about 880. The detection performance is still acceptable. It is due to its salient visual feature. Thus, the threshold varies with the class. Additionally, for the incomplete class, the performance declines with downsampling.


## 3.1.1. Annotation Quality of an Image

For the clarity of the description of annotation quality, an image is selected for evaluation, which is given in Figure2. The semantic hierarchy tree we defined is presented in Figure 3. The quality evaluation results for an image are given in Table 1. The accuracy of the bounding box for each object is shown in Figure 4.Figure 2
Human and ground truth annotations from the UTS dataset (ground truth and human annotations are shown in red and yellow, respectively).Figure 3
Semantic hierarchy tree and completeness of category of ground truth annotations for the original UTS training dataset.Table 1
Results of other quality dimensions for an image.
Annotation quality dimension
Value
Completeness of bounding box’s quantity
1
Completeness of bounding box’s size
0.64
Accuracy of bounding box
0.67
Consistency of bounding box
1
Completeness of bounding box’s label
1
Flat precision/recall of label
–/0.5
Hierarchical precision/recall of label
0.69/0.79
Flat precision/recall of hatchback
0.25/1
Hierarchical precision/recall of hatchback
0.42/0.83
Consistency of label
0.84Figure 4
Accuracy of the bounding box in an image (note that two small objects are missed by the image instance segmentation algorithm.).Now, the analysis is given below. According to Table1, the flat precision of hatchback is 0.25. However, it is because of the quality problems of ground truth annotations. Reviewing the annotations, we find that there are two small unannotated objects as shown in Figure 2. Hierarchical measures can reflect the relation of the classes. For instance, hierarchical precision for the hatchback is 0.42, while the flat precision is 0.25. Further, the consistency of the label is less than 1. It shows that there are inconsistent labels with the labels in adjacent frames. In Table 1, four metrics are equal to 1, reflecting that there is no error from these aspects.


## 3.1.2. Annotation Quality of Human and Ground Truth Annotations

Afterward, we show the annotation quality of the UTS dataset for the human and ground truth annotations. The annotation accuracies of the label are given in Tables2 and 3. The completeness of the category of the ground truth annotations for each class and the original vehicle dataset is given in Figure 3, where the threshold is set to 1000. The results of other quality dimensions are presented in Table 4.Table 2
Annotation accuracy of human annotations for the downsampled UTS dataset.
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Hatchback
0.79 ± 0.02
0.55 ± 0.14
0.92 ± 0.01
0.81 ± 0.07
Sedan
0.58 ± 0.08
0.78 ± 0.12
0.86 ± 0.03
0.88 ± 0.07
Minibus
0.93 ± 0.10
0.57 ± 0.33
0.95 ± 0.06
0.70 ± 0.22
SUV
0.20 ± 0.06
0.27 ± 0.10
0.69 ± 0.03
0.74 ± 0.04
MPV
0.19 ± 0.14
0.26 ± 0.14
0.62 ± 0.14
0.72 ± 0.08
Pickup
0.57 ± 0.41
1 ± 0
0.72 ± 0.27
1 ± 0
On average
0.55 ± 0.11
0.57 ± 0.07
0.79 ± 0.07
0.81 ± 0.05Table 3
Annotation accuracy of ground truth annotations for the downsampled UTS dataset.
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Hatchback
0.57 ± 0.13
0.57 ± 0.04
0.85 ± 0.06
0.67 ± 0.05
Sedan
0.79 ± 0.12
0.44 ± 0.04
0.91 ± 0.08
0.65 ± 0.02
Minibus
0.58 ± 0.34
0.77 ± 0.17
0.72 ± 0.23
0.79 ± 0.17
SUV
0.27 ± 0.10
0.15 ± 0.05
0.75 ± 0.04
0.53 ± 0.09
MPV
0.26 ± 0.14
0.17 ± 0.15
0.72 ± 0.08
0.51 ± 0.18
Pickup
1 ± 0
0.28 ± 0.19
1 ± 0
0.28 ± 0.19
On average
0.58 ± 0.07
0.40 ± 0.07
0.83 ± 0.04
0.57 ± 0.07Table 4
Results of other quality dimensions for the downsampled UTS dataset.
Annotation quality dimension
Human annotations
Ground truth annotations
Completeness of bounding box’s quantity
0.98 ± 0.02
0.75 ± 0.05
Completeness of bounding box’s size
0.96 ± 0.02
0.99
Consistency of bounding box
0.977 ± 0.002
0.96
Completeness of bounding box’s label
0.52 ± 0.11
0.58
Consistency of label
0.86 ± 0.01
0.71The quality of human annotations is analyzed first. According to Tables2 and 4, the overall annotation quality of the bounding box is good, while the annotation quality of the label is relatively poor. Accordingly, it can be inferred that the label’s annotation is a more difficult task. In particular, for SUV and MPV, the accuracy and recall are too low. The hierarchical accuracy is higher than the flat accuracy, treating errors distinctively. According to Table 4, compared with other dimensions, the consistency of the label is lower on account of the own property.The quality of ground truth annotations is evaluated here. According to Tables2–4, the completeness of bounding box’s quantity, flat and hierarchical recall of label, and consistency of label for ground truth annotations are lower than those for human annotations. When reviewing ground truth annotations, we find that ground truth annotations neglect some small and incomplete objects. But these small and incomplete objects can be annotated properly by experience. There are more inconsistent labels in ground truth annotations than in human annotations. Figure 3 shows that the completeness of category for MPV and pickup is 0, as the corresponding category’s quantities do not reach the threshold. Generally, the quality problem exists in the ground truth annotations. Therefore, it is significant to perform a quality assessment in the process of annotation and ground truth inference.


## 3.1.3. Relationship between the Completeness of Category and Detection Performance

For the sake of exploring the relationship between the completeness of category and detection performance, the following experiment is conducted, which implies the effectiveness of the dimension. The object detection experiment on the UTS dataset is performed on the original dataset and downsampled dataset. As for downsampling, we just select images for every two images. The detection algorithm we use is Faster RCNN [3]. Table 5 presents the corresponding result.Table 5
Comparison of detection results based on the original training dataset and downsampled dataset.
Class
Object number in the training dataset
mAP (original)
mAP (downsampled)
Hatchback
12165
0.669
0.744
Sedan
5484
0.573
0.565
Minibus
3220
0.663
0.601
SUV
1761
0.560
0.576
MPV
898
0.154
0.142
Pickup
263
0.020
0.0001
On average
3965.2
0.440
0.438According to Table5, we argue that the detection result is closely related to the completeness of category. Overall, for the complete class whose training samples’ quantity is over 1000, the corresponding mAP is high, while the detection mAPs of other classes are quite low. However, for SUV in the downsampled dataset, the quantity is about 880. The detection performance is still acceptable. It is due to its salient visual feature. Thus, the threshold varies with the class. Additionally, for the incomplete class, the performance declines with downsampling.


## 3.2. Case Study for PASCAL VOC 2007 Detection Dataset

In the case study, PASCAL VOC 2007 detection dataset is utilized for verification. To save labor, we select twenty images for each class as annotation samples. Finally, a random-selected dataset containing 353 images is obtained. The PASCAL VOC 2007 dataset is an image dataset. Consequently, a few quality dimensions are not fit for the dataset.


### 3.2.1. Annotation Quality for Human and Ground Truth Annotation

The quality of human and ground truth annotations for the PASCAL VOC 2007 dataset is given below. Accuracies of the label for the human and ground truth annotations are given in Tables6 and 7. The semantic hierarchy tree and completeness of category quantity are given in Figure 5, where the threshold is set as 400. The results of other quality dimensions are provided in Table 8.Table 6
Annotation accuracy of human annotations for the selected images of the PASCAL 2007 dataset (the average values are computed for the twenty classes).
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Person
0.98 ± 0.01
0.92 ± 0.05
0.99 ± 0.01
0.92 ± 0.04
Car
0.99 ± 0.02
0.94 ± 0.04
0.99 ± 0.01
0.94 ± 0.03
Chair
0.96 ± 0.03
0.74 ± 0.08
0.98 ± 0.01
0.82 ± 0.07
Bottle
0.98 ± 0.01
0.81 ± 0.08
0.99 ± 0.01
0.82 ± 0.07
Potted plant
1 ± 0
0.54 ± 0.31
1 ± 0
0.57 ± 0.29
Cow
0.99 ± 0.01
0.95 ± 0.01
0.997 ± 0.005
0.96 ± 0.01
Dining table
0.75 ± 0.16
0.59 ± 0.18
0.91 ± 0.06
0.64 ± 0.15
Bus
1 ± 0
0.94 ± 0.04
1 ± 0
0.96 ± 0.03
On average
0.96 ± 0.01
0.89 ± 0.04
0.983 ± 0.005
0.90 ± 0.04Table 7
Annotation accuracy of ground truth annotations for the selected images of the PASCAL 2007 dataset (the average values are computed for the twenty classes).
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Person
0.97 ± 0.01
0.79 ± 0.11
0.98 ± 0.01
0.81 ± 0.1
Car
0.94 ± 0.02
0.82 ± 0.08
0.96 ± 0.01
0.83 ± 0.08
Chair
0.90 ± 0.02
0.81 ± 0.09
0.96 ± 0.01
0.84 ± 0.08
Bottle
1 ± 0
0.74 ± 0.10
1 ± 0
0.81 ± 0.09
Potted plant
0.98 ± 0.02
0.77 ± 0.02
0.99 ± 0.01
0.78 ± 0.03
Cow
0.99 ± 0.01
0.81 ± 0.10
1 ± 0
0.83 ± 0.10
Dining table
0.68 ± 0.16
0.63 ± 0.11
0.87 ± 0.06
0.65 ± 0.09
Bus
0.91 ± 0.09
0.89 ± 0.08
0.94 ± 0.05
0.90 ± 0.07
On average
0.94 ± 0.02
0.84 ± 0.05
0.97 ± 0.01
0.87 ± 0.05Figure 5
Semantic hierarchy tree and completeness of category for original PASCAL VOC 2007 training dataset.Table 8
Results of other quality dimensions for the selected images and its original training dataset of the PASCAL VOC 2007 dataset.
Annotation quality dimension
Human annotations
Ground truth annotations
Completeness of bounding box’s quantity
0.90 ± 0.04
0.88 ± 0.05
Completeness of bounding box’s size
0.84 ± 0.02
0.85
Completeness of bounding box’s label
0.9991 ± 0.0008
1According to Tables6 and 8, we can see that the human annotation quality for the dataset is good overall. However, the accuracies of the chair, potted plant, and dining table are relatively poor. For instance, the average flat recall for the potted plant is 0.54. This is because the potted plant is small and tends to be neglected. And for the other dimensions of human annotations, quality is relatively reliable.Afterward, we evaluate the annotation quality of ground truth annotations. According to Tables6–8, we find that the quality of ground truth annotations is slightly worse than that of human annotations. Specifically, the completeness of the bounding box’s quantity and the flat recall of the label are relatively low. These dimensions indicate that there are more unannotated objects. As there are not enough images in the random-selected dataset, we calculate the completeness of category according to the original training set. The total completeness of category is 0.62, as 38% of the classes do not have enough samples.


### 3.2.2. Relationship between the Completeness of Category and Detection Performance

To explore the relationship between the completeness of category and detection performance, an experiment is conducted in the same way as the previous section. We conduct object detection experiments on the original dataset and downsampled dataset of which the sampling ratio is 0.5. And the major classes of person, car, and chair are not downsampled. Table9 presents the detection results, where classes are in descending order of quantity of training samples.Table 9
Comparison of detection results based on the original training dataset and downsampled dataset (the average values are computed for the twenty classes).
Class
Object number in the training dataset
mAP (original)
mAP (downsampled)
Person
5447
0.779
0.778
Car
1644
0.831
0.807
Chair
1432
0.520
0.511
Bottle
634
0.576
0.519
Potted plant
625
0.459
0.376
Cow
356
0.767
0.721
Dining table
310
0.682
0.671
Bus
272
0.772
0.776
On average
783.1
0.714
0.678According to Table9, on the whole, the detection performance declines after the dataset is downsampled. For the majority classes of person, car, and chair, there are no obvious declines of mAPs, as we do not make downsampling on these classes. As for the minority classes, mAPs for the bottle and potted plant decline a lot, which can be regarded as hard classes. But mAPs for the other classes of the minority are relatively high and change little, which should be regarded as easy classes. The hard classes are usually of small scale and have nonsalient visual features, hindering the learning of the object detection model. Therefore, the threshold for hard classes is relatively high. In the future process of constructing a dataset, the training samples’ quantity for hard classes should be added.


## 3.2.1. Annotation Quality for Human and Ground Truth Annotation

The quality of human and ground truth annotations for the PASCAL VOC 2007 dataset is given below. Accuracies of the label for the human and ground truth annotations are given in Tables6 and 7. The semantic hierarchy tree and completeness of category quantity are given in Figure 5, where the threshold is set as 400. The results of other quality dimensions are provided in Table 8.Table 6
Annotation accuracy of human annotations for the selected images of the PASCAL 2007 dataset (the average values are computed for the twenty classes).
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Person
0.98 ± 0.01
0.92 ± 0.05
0.99 ± 0.01
0.92 ± 0.04
Car
0.99 ± 0.02
0.94 ± 0.04
0.99 ± 0.01
0.94 ± 0.03
Chair
0.96 ± 0.03
0.74 ± 0.08
0.98 ± 0.01
0.82 ± 0.07
Bottle
0.98 ± 0.01
0.81 ± 0.08
0.99 ± 0.01
0.82 ± 0.07
Potted plant
1 ± 0
0.54 ± 0.31
1 ± 0
0.57 ± 0.29
Cow
0.99 ± 0.01
0.95 ± 0.01
0.997 ± 0.005
0.96 ± 0.01
Dining table
0.75 ± 0.16
0.59 ± 0.18
0.91 ± 0.06
0.64 ± 0.15
Bus
1 ± 0
0.94 ± 0.04
1 ± 0
0.96 ± 0.03
On average
0.96 ± 0.01
0.89 ± 0.04
0.983 ± 0.005
0.90 ± 0.04Table 7
Annotation accuracy of ground truth annotations for the selected images of the PASCAL 2007 dataset (the average values are computed for the twenty classes).
Class
Flat accuracy of label
Hierarchical accuracy of label
Precision
Recall
Precision
Recall
Person
0.97 ± 0.01
0.79 ± 0.11
0.98 ± 0.01
0.81 ± 0.1
Car
0.94 ± 0.02
0.82 ± 0.08
0.96 ± 0.01
0.83 ± 0.08
Chair
0.90 ± 0.02
0.81 ± 0.09
0.96 ± 0.01
0.84 ± 0.08
Bottle
1 ± 0
0.74 ± 0.10
1 ± 0
0.81 ± 0.09
Potted plant
0.98 ± 0.02
0.77 ± 0.02
0.99 ± 0.01
0.78 ± 0.03
Cow
0.99 ± 0.01
0.81 ± 0.10
1 ± 0
0.83 ± 0.10
Dining table
0.68 ± 0.16
0.63 ± 0.11
0.87 ± 0.06
0.65 ± 0.09
Bus
0.91 ± 0.09
0.89 ± 0.08
0.94 ± 0.05
0.90 ± 0.07
On average
0.94 ± 0.02
0.84 ± 0.05
0.97 ± 0.01
0.87 ± 0.05Figure 5
Semantic hierarchy tree and completeness of category for original PASCAL VOC 2007 training dataset.Table 8
Results of other quality dimensions for the selected images and its original training dataset of the PASCAL VOC 2007 dataset.
Annotation quality dimension
Human annotations
Ground truth annotations
Completeness of bounding box’s quantity
0.90 ± 0.04
0.88 ± 0.05
Completeness of bounding box’s size
0.84 ± 0.02
0.85
Completeness of bounding box’s label
0.9991 ± 0.0008
1According to Tables6 and 8, we can see that the human annotation quality for the dataset is good overall. However, the accuracies of the chair, potted plant, and dining table are relatively poor. For instance, the average flat recall for the potted plant is 0.54. This is because the potted plant is small and tends to be neglected. And for the other dimensions of human annotations, quality is relatively reliable.Afterward, we evaluate the annotation quality of ground truth annotations. According to Tables6–8, we find that the quality of ground truth annotations is slightly worse than that of human annotations. Specifically, the completeness of the bounding box’s quantity and the flat recall of the label are relatively low. These dimensions indicate that there are more unannotated objects. As there are not enough images in the random-selected dataset, we calculate the completeness of category according to the original training set. The total completeness of category is 0.62, as 38% of the classes do not have enough samples.


## 3.2.2. Relationship between the Completeness of Category and Detection Performance

To explore the relationship between the completeness of category and detection performance, an experiment is conducted in the same way as the previous section. We conduct object detection experiments on the original dataset and downsampled dataset of which the sampling ratio is 0.5. And the major classes of person, car, and chair are not downsampled. Table9 presents the detection results, where classes are in descending order of quantity of training samples.Table 9
Comparison of detection results based on the original training dataset and downsampled dataset (the average values are computed for the twenty classes).
Class
Object number in the training dataset
mAP (original)
mAP (downsampled)
Person
5447
0.779
0.778
Car
1644
0.831
0.807
Chair
1432
0.520
0.511
Bottle
634
0.576
0.519
Potted plant
625
0.459
0.376
Cow
356
0.767
0.721
Dining table
310
0.682
0.671
Bus
272
0.772
0.776
On average
783.1
0.714
0.678According to Table9, on the whole, the detection performance declines after the dataset is downsampled. For the majority classes of person, car, and chair, there are no obvious declines of mAPs, as we do not make downsampling on these classes. As for the minority classes, mAPs for the bottle and potted plant decline a lot, which can be regarded as hard classes. But mAPs for the other classes of the minority are relatively high and change little, which should be regarded as easy classes. The hard classes are usually of small scale and have nonsalient visual features, hindering the learning of the object detection model. Therefore, the threshold for hard classes is relatively high. In the future process of constructing a dataset, the training samples’ quantity for hard classes should be added.


## 4. Conclusion

Annotation quality is essential for the object detection model’s training. In this paper, conceptual cognitive modeling for fine-grained annotation quality assessment is proposed. The annotation quality is calculated from the perspectives of the bounding box and label. To begin with, a generic framework based on general-purpose data quality dimensions is constructed from two aspects: the bounding box and the class label. This framework is used to assess the completeness and accuracy from the corresponding aspects. Nonetheless, the basic framework has limitations in assessing the consistency, the category’s quantity, and the annotation errors. Thereupon, the cognitive theory is introduced, and we add the corresponding elements, including consistency of bounding box, hierarchical accuracy of label, consistency of label, and completeness of category. Case studies on the Urban Traffic Surveillance dataset and PASCAL VOC 2007 detection dataset indicate the validity of the framework. Currently, the annotation quality framework is constructed in an ideal condition. Future research is required to consider more practical factors.



---

*Source: 6195189-2020-05-05.xml*