Practitioner’s guide to IoU, Non-Max suppression, and Mean Average Precision
By: Vijayabhaskar J
Tags: AI, Machine Learning, Artificial Intelligence, Object Detection, Deep Learning

Implementations from scratch in Pytorch for Object Detection

This article is aimed at an audience who likes to look at the code behind the algorithms to understand how they work instead of reading the math behind them.

Intersection over Union (IoU)

Intersection over Union(IoU) also knows as Jaccard’s index is literally calculating the “Area of Intersection” of two boxes over the “Area of Union” of the same two boxes. In object detection you IoU is calculated between the ground truth bounding box and the predicted bounding boxes to estimate how close our predicted bounding box is to the ground truth.

IoU of 1 means your predicted bounding box perfectly matches the ground truth box.

IoU of 0 means no part of your predicted bounding box overlaps with the ground truth box.

Non-max Suppression

Typically an object detection algorithm produces multiple bounding boxes for the same object in an image, Non-max Suppression is a way to remove these duplicate detections.

Non-max Suppression can be expressed as a series of steps:

Every object detection algorithm outputs object confidence scores to denote how confident the detector about that particular bounding box prediction.

Step 1:

Discard all the predicted boxes whose confidence scores are less than a user-defined threshold.

Step 2:

While there are any remaining predicted bounding boxes in the list of predicted bounding boxes:

Pick the bounding box with the largest confidence score and output it as a prediction.

Compare this IoU of this bounding box with every other predicted bounding box of the same class, and if the IoU is greater than the user-defined IoU threshold, discard it as it’s duplicate detection.

Remove the outputted predicted bounding box from the list of bounding boxes.

Mean Average Precision (mAP)

Mean Average Precision is a metric used to measure the accuracy of the predictions, it is simply the average of “Average Precision” calculated across all classes.

Average Precision(AP) can be defined as the Area Under the Precision-Recall Curve. To plot the Precision-Recall curve we need to define what is True Positive, False Positive, True Positive, and True Negative in the sense of object detection.

True Positive: If the IoU of a predicted bounding box of class ‘c’ and the ground truth box of the same class ‘c’ is ≥ threshold.

False Positive: If the IoU of a predicted bounding box of class ‘c’ and the ground truth box of the same class ‘c’ is < threshold.

Even if the IoU> threshold, if any detection is already detected as a TP for the current ground truth bounding box then this should be labeled as “FP”(as applied by the PASCAL VOC 2012 challenge).

Even if the IoU> threshold, if any detection is already detected as a TP for the current ground truth bounding box then this should be labeled as “FP”(as applied by the PASCAL VOC 2012 challenge). True Negative: This is not applicable to object detection as this would mean there is no ground truth box and the model didn’t predict it as a bounding box, there are infinite possible positions to satisfy this.

False Negative: Undetected ground truth bounding box.

Precision and recall can be calculated as,

In practice, the True Positive(TP), False Positive(FP) are enough to calculate the Precision and Recall, as Recall can be directly calculated as TP/len(ground_truths).

The TP and FP here are cumulatively summed across all detections which are sorted by the confidence scores to get a list of TPs and FPs for the Precision-Recall graph.

Once the TP_cumsum and FP_cumsum were calculated, the Average Precision can be calculated in many ways, it is simply the AUC of the Precision x Recall curve. But if you plot the curve you can see a zigzag pattern.

This makes calculating AUC less accurate, there are two popular approaches to overcome this. They are:

11 point interpolation

All point interpolation

11 point interpolation

This is the way PASCAL VOC competition used to evaluate objection detection algorithms.

Simply explained:

“Average” of the maximum “precision” values at 11 equally distributed points between 0 and 1, where recall is ≥ to these 11 points.

This is so simple, that it is better explained in code itself.

Example:

Let's consider this data:

Plot:

Averaging all max precision(red dots) gives you the 11 points interpolated Average precision.

AP = (1+0.6666 + 0.4285 + 0.4285 + 0.4285+0+0+0+0+0+0)/11

AP = 0.2684 or 26.84%

All point interpolation

In all point interpolation, instead of considering max precision at 11 fixed points, all points where recall value changes are considered. For each pair of points in recall change, the distance is calculated. For example If recall changes from 0.1 to 0.25 to 0.33, the distances will be 0.15 and 0.08. These distances can be thought of width of a rectangle, where the height is the max precision at these respective recall positions. With height and width in hand, the area at this part of the precision-recall curve can be calculated. Repeating this for every recall change will cover the entire curve.

This is also better explained with code:

Example:

Plot:

In this plot, For each pair of green dots calculate the x-axis distance between them and multiply it with the max precision in the region to get the area in that region.

Sum all the areas you get the Area under the curve (Average Precision):

AP = 1 ∗ (0.0666–0) + 0.6666 ∗ (0.1333–0.0666) + 0.4285 ∗ (0.4–0.1333) + 0.3043 ∗ (0.4666–0.4)

AP = 0.2456 or 24.56%

Finally, complete code to calculate Mean Average Precision(mAP):

COCO competition used variations of mAP.

From “ A Survey on Performance Metrics for Object-Detection Algorithms ” paper by Rafael Padilla, Sergio L. Netto, Eduardo A. B. da Silva:

The COCO detection challenge (bounding box) is a competition which provides bounding-box coordinates of more than 200,000 images comprising 80 object categories. The submitted works are ranked according to metrics gathered into four main groups.

AP: The AP is evaluated with different IOUs. It can be calculated for 10 IOUs varying in a range of 50% to 95% with steps of 5%, usually reported as AP@50:5:95. It also can be evaluated with single values of IOU, where the most common values are 50% and 75%, reported as AP50 and AP75 respectively.

AP Across Scales: The AP is determined for objects in three different sizes: small (with area < 322 pixels), medium (with 322 < area < 962 pixels), and large (with area > 962 pixels).

Average Recall (AR): The AR is estimated by the maximum recall values given a fixed number of detections per image (1, 10 or 100) averaged over IOUs and classes.

AR Across Scales: The AR is determined for objects in the same three different sizes as in the AP Across Scales, usually reported as AR-S, AR-M, and AR-L, respectively.

References: