Introduction to Object Detection Model Evaluation | by Déborah Mesquita | Towards Data Science

Evaluating object detection models is not straightforward because each image can have many objects and each object can belong to different classes. This means that we need to measure if the model found all the objects and also a way to verify if the found objects belong to the correct class.
This means that an object detection model needs to accomplish two things:
Find all the objects in an image
Check if the found objects belong to the correct class
In this article you’ll see how the object detection researchers created ways to combine these two things in a single metric, the mean Average Precision (mAP).
Two recent open-source tools were released recently and they help us a lot on these evaluation tasks. This means that right now is a great time to start working on object detection models. Let’s get started!
When you’re annotating objected, the bounding box are the boxes you draw to indicate the location.
A object detection model produces the output in three components:
The bounding boxes — x1, y1, width, height if using the COCO file format
The class of the bounding box
The probability score for that prediction— how certain the model is that the class is actually the predicted class
We need to take a closer look at the probability score component. Mostly because modern neural networks are poorly calibrated [1]:
Recent advances in neural network architecture and training — model capacity, normalization, and regularization– have strong effects on network calibration — Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, On Calibration of Modern Neural Networks
Now imagine you need to compare the performance of two different object detection models. Since we know that the probability scores are poorly calibrated, there is a good chance that a bounding box with a probability score of say 80% having the same location coordinates as a bounding box with a probability score of 50% on the other model. How would we go about comparing the performance of these models? We’ll figure this out in the next sections.
As you can see, the object detection task introduces some new challenges that we don’t see in regular classification tasks. Let’s talk about three of them in the next sections.
In a multiclass classification task, you know the distribution of your dataset per image because each image can only have one class. This means you can easily split your training/test datates. But in object detection, each image can have many objects from different classes.
Imagine that you have a dataset with images from supermarket shelves. Given that you don’t have access to new data and you need to split this dataset to create a test set, how would you split the images into the training/test groups?
In a project I was working on we did some experiments with clustering using the brands and the quantity of products to get a sample from the images. We still need to do more research to present the findings but it seems to provide us with a good proxy to create the training/test groups in object detection dataset.
To evaluate if an object was located we use the Intersection over Union (IoU) as a similarity measure. It’s given by the area of the overlap divided by the size of the union of the two bounding boxes.
The IoU is needed to calculate the Average Precision because the first step is to define if two bounding boxes are pointing to the same object. To answer this question the default IoU value we use is 0.5.
The model outputs the coordinates of the bounding boxes and the probability score for the predicted class. The problem is that the neural networks are poorly calibrated, meaning that we don’t have a direct way of comparting the probability scores of two different models.
The solution for this problem is using the ranking of the bounding boxes each model outputs. This is the way the Average Precision (AP) metric is calculated. You start sorting the probability scores and then check if the bounding box is a True Positive or a False Positive using an IoU threshold to compare them.
As Harshit Kumar [2] points out in this great explanation, if there is more than one detection for a single object, the detection having the highest IoU is considered as TP, rest as FP.
You count the accumulated TP and the accumulated FP and compute the precision/recall at each line. Usually, the Average Precision is computed as the average precision at 11 equally spaced recall levels. The Mean Average Precision (mAP) is the averaged AP over all the object categories, and it’s the metric more used to evaluate an object detection model.
Since we first rank the predictions, now the absolute probability scores don’t matter much and we can use the same metric for different models.
One peculiarity of object annotation tasks is that there is a lack of consensus on the representation of the location and size of objects and performance assessment tools implement different metrics [3].
If you need to evaluate an object detection model and don’t want to compute the metrics from scratch (why would you like to do that? haha) the authors of [3] developed and released a tool named Open-Source Toolbox for Object Detection Metrics. The toolkit fills the gap of a reliable source of object detection metrics.
The mAP is good if we need to compare models, but what if our goal is to have a qualitative view of how it’s performing? We’ll talk about this in the next section.
The mAP is good for a competition settings or to get a quick assessment of how a model is performing, but if you care about understanding what the model is doing you’ll need different metrics.
The source of errors in object detection tasks can be various:
the model could find the correct location of the bounding box but output it with an incorrect class
the model could find an object with a correct class but be totally wrong about the location of the bounding box
the model could miss an object entirely
and so on
To solve this problem the authors of [4] developed TIDE. TIDE is a general toolbox to compute and evaluate the effect of object detection and instance segmentation on overall performance.
The toolbox determines the importance of a given error to overall to the mAP by fixing the error and observing the resulting change in mAP. Let’s see what kind of errors are analyzed and how they do it.
The authors define 6 error types:
Classification Error: localized correctly but classified incorrectlyLocalization Error: classified correctly but localized incorrectlyBoth Cls and Loc Error: classified incorrectly and localized incorrectlyDuplicate Detection Error: would be correct if not for a higher scoring detectionBackground Error: detected background as fore-groundMissed GT Error: All undetected ground truth (false negatives) not already covered by classification or localization error
Classification Error: localized correctly but classified incorrectly
Localization Error: classified correctly but localized incorrectly
Both Cls and Loc Error: classified incorrectly and localized incorrectly
Duplicate Detection Error: would be correct if not for a higher scoring detection
Background Error: detected background as fore-ground
Missed GT Error: All undetected ground truth (false negatives) not already covered by classification or localization error
To measure each type of error an oracle is implemented to correct them and the tool computes how the mAP would increase if this type of error didn’t exist. This information is gold because it can guide us on how to improve the model.
Here is an example of TIDE results:
-- mask_rcnn_bbox --bbox AP @ 50: 61.80                         Main Errors=============================================================  Type      Cls      Loc     Both     Dupe      Bkg     Miss-------------------------------------------------------------   dAP     3.40     6.65     1.18     0.19     3.96     7.53=============================================================        Special Error=============================  Type   FalsePos   FalseNeg-----------------------------   dAP      16.28      15.57=============================
We can see that you could increase the mAP by 7.53 points if you correct the Missed GT Error. Now you can investigate the training images and the validation images to figure out how to fix these errors. This is something we’ll probably need a lot of time to find out if we were working only with the mAP result, 61.80.
Object Detection tasks come with challenges we don’t see in multiclass classification tasks. With object detection, you need to find the objects and predict the correct class. Also, each image usually have a variable number of annotated objects.
Fortunately, a lot of good researchers have been working on these problems and noew we have ways to work with them. To compare two bounding boxes are similar we use the Intersection over Union (IoU) and to measure the model performance we use the mean Average Precision (mAP). The Open-Source Toolbox for Object Detection Metrics provides an open-source tool that supports many bounding box formats and evaluates detections with different metrics.
Having just the mAP is not enough to figure out what you need to do to increase the performance of a model. To solve that problem the authors of [4] created TIDE, a toolbox to compute and evaluate the effect of object detection on overall model performance.
[1] Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, “On Calibration of Modern Neural Networks” arXiv:1706.04599 [cs.LG], Aug. 2016.
[2] Kumar, Harshit. “Evaluation metrics for object detection and segmentation: mAP”. https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation
[3] Padilla, Rafael, et al. “A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit.” Electronics 10.3 (2021): 279.
[4] Bolya, Daniel, et al. “Tide: A general toolbox for identifying object detection errors.” arXiv preprint arXiv:2008.08115 (2020).