Title: Mean Average Precision (mAP) Using the COCO Evaluator - PyImageSearch

URL Source: https://pyimagesearch.com/2022/05/02/mean-average-precision-map-using-the-coco-evaluator/

Published Time: 2022-05-02T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Mean Average Precision (mAP) Using the COCO Evaluator](#H2BlogTitle)

*   [Configuring Your Development Environment](#H3Config)
*   [Having Problems Configuring Your Development Environment?](#H3Hav)
*   [Project Structure](#H3Proj)

*   [Intersection over Union (IoU)](#H2IoU)

*   [Why Do We Use IoU?](#H3Why)
*   [Coding IoU in Python](#H3Coding)

*   [Precision and Recall](#H2P-R)

*   [Precision](#H3Precision)
*   [Recall](#H3Recall)
*   [TP, FP, TN, and FN in Detection Context](#H3Detection)

*   [True Positive](#H4TP)
*   [False Positive](#H4FP)
*   [True Negative](#H4TN)
*   [False Negative](#H4FN)

*   [Precision-Recall Curve](#H3Curve)
*   [Object Detection Competition Datasets](#H3Competition)

*   [PASCAL VOC 2012](#H4PASCAL)
*   [MS COCO](#H4MSCOCO)

*   [Precision-Recall Curve for Object Detection](#H3Object)
*   [Calculating the Average Precision](#H3Calc)

*   [11-Point Interpolation](#H4-11-Point)

*   [Mean Average Precision (mAP)](#H3Mean)

*   [Evaluating the YOLOv4 Model Using the COCO Evaluator](#H2Eval)

*   [COCO mAP Evaluator](#H3COCOmAP)
*   [Configuring the Prerequisites](#H3Prereq)

*   [Summary](#H2Summary)

*   [Citation Information](#H3CitInfo)

* * *

In this tutorial, you will learn Mean Average Precision (mAP) in object detection and evaluate a YOLO object detection model using a COCO evaluator.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)

This is the 4th lesson in our 7-part series on the **YOLO Object Detector**:

1.  [_Introduction to the YOLO Family_](https://pyimg.co/dgbvi)
2.  [_Understanding a Real-Time Object Detection Network: You Only Look Once (YOLOv1)_](https://pyimg.co/3cpmz)
3.  [_A Better, Faster, and Stronger Object Detector (YOLOv2)_](https://pyimg.co/ehaox)
4.  [_Mean Average Precision (mAP) Using the COCO Evaluator_](https://pyimg.co/nwoka) **(today’s tutorial)**
5.  [_An Incremental Improvement with Darknet-53 and Multi-Scale Predictions (YOLOv3)_](https://pyimg.co/8xfpg)
6.  [_Achieving Optimal Speed and Accuracy in Object Detection (YOLOv4)_](https://pyimg.co/c6kiu)
7.  [**_Training the YOLOv5 Object Detector on a Custom Dataset_**](https://pyimg.co/fq0a3)

We will discuss these key topics in this post:

*   What are Precision and Recall?
*   The Precision-Recall (PR) curve
*   Intersection over Union (IoU)
*   Average Precision (AP)
*   Mean Average Precision (mAP)
*   Walk through the code implementation of evaluating a YOLO object detection model using a COCO evaluator

Before we get started, are you familiar with how an object detector works, especially a single-stage detector like YOLO?

If not, be sure to look at our previous posts, [Introduction to the YOLO Family](https://pyimg.co/dgbvi) and [Understanding a Real-Time Object Detection Network: You Only Look Once (YOLOv1)](https://pyimg.co/3cpmz), for a high-level intuition of how a single-stage object detection works in general. Single-Stage Object Detectors treat object detection as a simple regression problem. For example, the input image fed to the network directly outputs the class probabilities and bounding box coordinates.

These models skip the region proposal stage, also known as Region Proposal Network, which is generally part of Two-Stage Object Detectors that are areas of the image that could contain an object.

**To learn what is mAP in object detection and how to evaluate an object detection model using a COCO evaluator,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

* * *

**[Mean Average Precision (mAP) Using the COCO Evaluator](#TOC)**
-----------------------------------------------------------------

When solving a problem involving machine learning and deep learning, we usually have various models to choose from; for example, in image classification, one could select VGG16 or ResNet50. Each one has its peculiarities and would perform differently based on various factors like the dataset or target platform. And to finally decide on the best model by objectively comparing models for our use case, we need to have an evaluation metric in place.

After the model is trained or fine-tuned on the training set, it is then judged by how well or accurately it performs over the validation and test data. Various evaluation metrics or statistics could evaluate the deep learning models, but which metric to use depends on the particular problem statement and application.

The most common metric used for evaluation in an image classification problem is _Precision_, _Recall_, _Confusion-matrix_, _PR-curve_, etc. While in image segmentation, Mean Intersection over Union, aka _mIoU_, is used.

However, if we address the elephant in the room, the most common metric of choice used for Object Detection problems is _Mean Average Precision_ (aka _mAP)_.

Since in object detection, the objective is not only to correctly classify the object (or objects) in the image but to also find where in the image it is located, we cannot simply use the image classification metrics like Precision and Recall.

Hence, the object detection evaluation metric needs to consider both the category and location of the objects in its formulation, and that’s where _mAP_ comes into play. And to understand _mAP_, it is necessary to understand _IoU_, _Precision_, _Recall_, and _Precision-Recall_ curve. So let’s dig right into it!

* * *

### **[Configuring Your Development Environment](#TOC)**

To follow this guide, you need to have the Pycocotools library installed on your system. For additional image handling purposes, you’ll be using OpenCV, Sklearn for computing Precision and Recall, Matplotlib for plotting graphs, and a few more libraries.

Luckily, all the libraries below are pip-installable!

$ pip install pycocotools==2.0.4
$ pip install matplotlib==3.2.2
$ pip install opencv==4.1.2
$ pip install progressbar==3.38.0
$ pip install sklearn==1.0.2

That is all for the environment configuration necessary for this guide!

However, you will also need to install the Darknet framework on your system to run an inference with YOLOv4. For that, we would highly recommend you check out [AlexeyAB’s Darknet Repository](https://github.com/AlexeyAB/darknet).

* * *

### **[Having Problems Configuring Your Development Environment?](#TOC)**

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://www.pyimagesearch.com/pyimagesearch-university/)

**Figure 1:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://www.pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://www.pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### **[Project Structure](#TOC)**

Before we get into the theory and implementation of various key topics, let’s look at the project structure.

Once we have downloaded our project directory, it should look like this:

$ tree .
.
├── darknet
├── data
│   ├── dog.jpg
│   ├── label2idx.pkl
│   └── yolo\_90\_class\_map.pkl
├── eval\_iou.py
├── eval\_map.py
├── main.py
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   ├── config.py
│   ├── utils\_iou.py
│   ├── utils\_map.py
│   └── utils\_precision\_recall.py
└── results
    ├── COCO\_Val\_mAP.png
    ├── dog\_iou.png
    └── pr\_curve.png

36 directories, 2135 files

Please note that we do not show the `darknet` directory since it has a lot of files, and it would be difficult to put all the files as part of the directory structure.

The parent directory has 3 python scripts and 4 folders.

*   `main.py`: This is the driver script that is the entry point to our experiments. Based on the command line input received from the user, it will execute one of three experiments (IoU, Precision-Recall Curve, or COCO Evaluator).

*   `eval_iou.py`: The python script has the `compute_iou()` method. It is called from the main. It takes `imagePath` as a parameter and calls two methods: first to compute IoU and second to display the output on the screen.

*   `eval_map.py`: It loads the ground-truth annotation file, calls the `run_inference()` method to generate the predictions, and then computes mAP and displays the results on the terminal.

Next, let’s get into the `pyimagesearch` directory! In it, we will find 5 python scripts:

*   `__init__.py`: This will make Python treat the `pyimagesearch` directory as a module
*   `config.py`: This script contains various hyperparameter presets and defines the data paths
*   `utils_iou.py`: This script has the implementation of the IoU method and plot method
*   `utils_map.py`: This script contains an `image_detection` method that runs inference per image, a few methods for loading pickle files, and modifying bounding box coordinates. It also has the `run_inference()` method that iterates over each image calls `image_detection()` and stores the prediction in JSON format
*   `utils_precision_recall.py`: This script has a bunch of methods for compute precision, recall over various thresholds, and a method to plot the precision-recall curve

Next, we have the `data` directory, which contains:

*   Sample image on which we will run the IoU experiment
*   Pickle files that store object category related information
*   We will download the YOLOv4 weight in this directory
*   Finally, we will also download the MS COCO validation images and ground-truth annotations here

Lastly, we have the `results` directory, which contains:

*   `COCO_Val_mAP.png`: The YOLOv4 mAP results on the COCO validation dataset
*   `dog_iou.png`: A sample IoU output on the dog image
*   `pr_curve.png`: The Precision-Recall curve plot

* * *

**[Intersection over Union (IoU)](#TOC)**
-----------------------------------------

Intersection over Union, also known as [Jaccard Index](https://en.wikipedia.org/wiki/Jaccard_index), measures the overlap between the ground-truth bounding boxes and the predicted bounding boxes. It is the ratio between the Intersection and Union of the ground truth boxes with the predicted bounding boxes. The _ground-truth bounding boxes_ are the hand-labeled bounding boxes from the testing set that specify _wherein_ the image of our object is and the predicted bounding boxes come from the model.

As long as we have these two (ground-truth and prediction) bounding boxes, we can apply Intersection over Union.

**Figure 2** is a visual example of a ground-truth bounding box versus a predicted bounding box. The goal is to compute the Intersection over Union between these two bounding boxes.

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20450'%3E%3C/svg%3E)

**Figure 2:** An example of detecting a stop sign in an image. The predicted bounding box is drawn in red, while the ground-truth bounding box is drawn in green ([source](https://pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/)).

In **Figure 2**, we can see that the object detector has detected the presence of a stop sign in an image. The _predicted_ bounding box is drawn in _red_, while the _ground-truth_ (i.e., hand-labeled) bounding box is drawn in green.

As shown in **Figure 3**, Intersection over Union is as simple as dividing the area of overlap between the bounding boxes by the area of union.

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20468'%3E%3C/svg%3E)

**Figure 3:** Intersection over Union (Area of Overlap/Area of Union) ([source](https://pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/)).

From **Figure 3**, you can see that Intersection over Union is simply a ratio. In the numerator, we compute the _area of overlap_ between the _predicted_ bounding box and the _ground-truth_ bounding box.

The denominator is the _area of union_ or the area encompassed by the predicted bounding box and the ground-truth bounding box.

The IoU score is normalized (since the denominator is the _union area_)_,_ ranging from 0.0 to 1.0. Here 0.0 represents no overlap between the predicted and ground-truth bounding box, while 1.0 is the most optimal, meaning the predicted bounding box fully overlaps the ground-truth bounding box.

**An Intersection over Union score** **_\> 0.5_** **is typically considered a “good” prediction.**

* * *

### **[Why Do We Use IoU?](#TOC)**

Suppose you have performed any previous machine learning in your career, specifically classification. In that case, you’ll likely be used to _predicting class labels_ where your model outputs a single label that is either _correct_ or _incorrect._

This type of binary classification makes computing accuracy straightforward; however, it’s not so simple for object detection.

In reality, it’s _implausible_ that the _(x, y)_\-coordinates of our predicted bounding box will _exactly match_ the _(x, y)_\-coordinates of the ground-truth bounding box.

Due to varying parameters of our model (image pyramid scale, sliding window size, feature extraction method, etc.), a complete and total match between predicted and ground-truth bounding boxes is simply unrealistic.

Because of this, we need to define an evaluation metric that _rewards_ predicted bounding boxes for heavily overlapping (not 100%) with the ground truth.

**Figure 4** shows examples of good and bad Intersection over Union scores; the IoU scores demonstrated in each case are just for intuition purposes and might not be precise.

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20248'%3E%3C/svg%3E)

**Figure 4:** IoU in three different scenarios ([source](https://pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/)).

As you can see, predicted bounding boxes that heavily overlap with the ground-truth bounding boxes have higher scores than those with less overlap. This makes Intersection over Union an excellent metric for evaluating custom object detectors.

We aren’t concerned with an _exact_ match of _(x, y)_\-coordinates, but we want to ensure that our predicted bounding boxes match as closely as possible — Intersection over Union can take this into account.

And soon, you will see how the concept and magic of IoU benefit the _mAP_ metric in an object detection model.

Before diving into the implementations of IoU, Precision-Recall Curve, and Evaluating YOLOv4 object detector, let’s set up our paths and hyperparameters. For that, we will hop into the module `config.py`. Most other scripts in our project will call this module and use its presets.

\# import the necessary packages
from glob import glob

YOLO\_CONFIG = "darknet/cfg/yolov4.cfg"
YOLO\_WEIGHTS = "data/yolov4.weights"
COCO\_DATA = "darknet/cfg/coco.data"

YOLO\_NETWORK\_WIDTH = 608
YOLO\_NETWORK\_HEIGHT = 608

LABEL2IDX = "data/label2idx.pkl"
YOLO\_90CLASS\_MAP = "data/yolo\_90\_class\_map.pkl"

IMAGES\_PATH = glob("data/val2017/\*")

COCO\_GT\_ANNOTATION = "data/annotations/instances\_val2017.json"
COCO\_VAL\_PRED = "data/COCO\_Val\_Predictions.json"

CONF\_THRESHOLD = 0.25

IOU\_GT = \[90, 80, 250, 450\]
IOU\_PRED = \[100, 100, 220, 400\]

IOU\_RESULT = "results/dog\_iou.png"
PR\_RESULT = "results/pr\_curve.png"

GROUND\_TRUTH\_PR = \["dog", "cat", "cat", "dog", "cat", "cat", "dog",
  "dog", "cat", "dog", "dog", "dog", "dog", "cat", "cat", "dog"\]
PREDICTION\_PR = \[0.7, 0.3, 0.5, 0.6, 0.3, 0.35, 0.55, 0.2, 0.4, 0.3,
  0.7, 0.5, 0.8, 0.2, 0.9, 0.4\]

In the above lines of code, we first import the `glob` module (on **Line 2**), which is used to get all the images path as listed on **Line 14**.

From **Lines 4-6**, we define the YOLOv4 model weights and configuration file and also the COCO data file. Then we define the YOLOv4 model input width and height (i.e., `608`).

Then, on **Lines 11 and 12**, we define the pickle file paths. The COCO ground-truth annotations and prediction JSON file paths are declared on **Lines 16 and 17**. The YOLOv4 confidence threshold is specified on **Line 19**, which is set to `0.25`.

On **Lines 21-24**, the IoU ground-truth and prediction box coordinates are defined along with the IoU result path.

Finally, from **Lines 25-30**, the Precision-Recall curve experiment-related parameters are defined.

* * *

### **[Coding IoU in Python](#TOC)**

Now that we understand what Intersection over Union is and why we use it to evaluate object detection models, let’s go ahead and implement it in the `utils_iou.py` script. Please note that we do not run an object detector to get the predicted bounding box coordinates for the IoU example; we assume predicted coordinates. For the ground-truth coordinates, we manually annotated the image.

\# import the necessary packages
from matplotlib import pyplot as plt
import cv2

def intersection\_over\_union(gt, pred):
    # determine the (x, y)-coordinates of the intersection rectangle
    xA = max(gt\[0\], pred\[0\])
    yA = max(gt\[1\], pred\[1\])
    xB = min(gt\[2\], pred\[2\])
    yB = min(gt\[3\], pred\[3\])

    # if there is no overlap between predicted and ground-truth box
    if xB < xA or yB < yA:
        return 0.0

    # compute the area of intersection rectangle
    interArea = max(0, xB - xA + 1) \* max(0, yB - yA + 1)

    # compute the area of both the prediction and ground-truth
    # rectangles
    boxAArea = (gt\[2\] - gt\[0\] + 1) \* (gt\[3\] - gt\[1\] + 1)
    boxBArea = (pred\[2\] - pred\[0\] + 1) \* (pred\[3\] - pred\[1\] + 1)

    # compute the intersection over union by taking the intersection
    # area and dividing it by the sum of prediction + ground-truth
    # areas - the intersection area
    iou = interArea / float(boxAArea + boxBArea - interArea)

    # return the intersection over union value
    return iou

We begin with the necessary imports in **Lines 2 and 3**.

On **Line 5**, we define the `intersection_over_union` method, which requires two parameters: `gt` and `pred` are presumed to be our ground-truth and predicted bounding boxes (the actual _order_ you supply these two parameters to `intersection_over_union` does not matter).

Then, from **Lines 7-10**, we determine the _(x, y)_\-coordinates of the intersection rectangle, which are then used to compute the intersection area on **Line 17**.

On **Lines 13 and 14**, we perform a sanity check that if there is zero overlap between the predicted and ground-truth bounding box, then we return zero.

The `inter_Area` variable represents the _numerator_ in the Intersection over Union formula.

To compute the denominator (i.e., the _union area_), we first need to derive the area of the predicted bounding box and the ground-truth bounding box (**Lines 21 and 22**).

**_Note:_** Since we are working in a “pixel coordinate space” to calculate the area of the rectangle, we _must_ add `1` in each direction. The first pixel would start from 0 till (height/width ![Image 7: -](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2011%201'%3E%3C/svg%3E) 1) in the pixel space. For example, if an image has a width of 1920, the pixels would start from `0` and go till `1919`, which would result in an incorrect area. Hence, we add `1` to the intersection and union areas calculation.

The Intersection over Union can then be computed on **Line 27** by dividing the intersection area by the union area of the two bounding boxes, taking care to subtract out the intersection area from the denominator (otherwise, the intersection area would be counted twice).

Finally, the Intersection over Union score is returned to the calling function on **Line 30**.

def plt\_imshow(title, image, path):
    # convert the image frame BGR to RGB color space and display it
    image = cv2.cvtColor(image, cv2.COLOR\_BGR2RGB)
    plt.imshow(image)
    plt.title(title)
    plt.grid(False)
    plt.axis("off")
    plt.imsave(path, image)
    plt.show()

From **Lines 32-40**, we define the `plt_imshow` method that accepts two arguments: the plot’s title and the image to be displayed. And it displays the output image using the `plt.show()` method. Since the `matplotlib` plot function expects the image to be in the RGB color space, we convert the image from BGR to RGB on **Line 34**.

Once we have written the method for computing Intersection over Union, it is time to wrap everything together in a `compute_iou` method where we will pass the `imagePath` on which we compute the IoU, so let’s get into the `eval_iou.py` script.

\# import the necessary packages
from pyimagesearch.utils\_iou import intersection\_over\_union
from pyimagesearch.utils\_iou import plt\_imshow
from pyimagesearch import config
import cv2

def compute\_iou(imagePath):
    # load the image
    image = cv2.imread(imagePath)

    # define the top-left and bottom-right coordinates of ground-truth
    # and prediction
    groundTruth = \[90, 80, 250, 450\]
    prediction = \[100, 100, 220, 400\]

    # draw the ground-truth bounding box along with the predicted
    # bounding box
    cv2.rectangle(image, tuple(groundTruth\[:2\]),
        tuple(groundTruth\[2:\]), (0, 255, 0), 2)
    cv2.rectangle(image, tuple(prediction\[:2\]),
        tuple(prediction\[2:\]), (0, 0, 255), 2)

    # compute the intersection over union and display it
    iou = intersection\_over\_union(groundTruth, prediction)
    cv2.putText(image, "IoU: {:.4f}".format(iou), (10, 34),
        cv2.FONT\_HERSHEY\_SIMPLEX, 1.4, (0, 255, 0), 3)

    # show the output image
    plt\_imshow("Image", image, config.IOU\_RESULT)

On **Line 9**, we load the `image` from the disk and then draw the ground-truth bounding box in green (**Lines 18 and 19**), followed by the predicted bounding box in red (**Lines 20 and 21**).

The Intersection over Union metric is computed on **Line 24** by passing in the ground-truth and predicted bounding box coordinates that we defined on **Lines 13 and 14**.

We then write the Intersection over Union value on the image (**Lines 25 and 26**), and finally, the output image is displayed on our screen (**Line 29)**.

In **Figure 5**, we can see that the Intersection over Union score is `0.6098`, indicating a significant overlap between the ground-truth (green) and predicted (red) bounding box.

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20338%20451'%3E%3C/svg%3E)

**Figure 5:** IoU depiction with ground-truth in green and proposed detection in red.

* * *

**[Precision and Recall](#TOC)**
--------------------------------

This section will discuss the precision, recall, and precision-recall curve. Since you are in the object detections series, I am sure that most of you would have worked on image classification problems earlier and would already be familiar with precision, recall, and confusion matrix and not feel alienated by these technical terms.

However, let’s recap these terms.

Imagine working on a binary classification problem in which each input sample is assigned to one of the two classes, that is, 1 (positive) and 0 (negative). For example, these two class labels could be spam or no-spam, malignant or benign (medical imaging – cancer classification problem), a simple cat or dog classifier, etc.

Let’s take an example to understand it in more detail. Assume we have a dataset with ten images of cats and others. We need to classify given an input sample where the class belongs. This means that it’s a binary classification problem.

Here we have the `ground-truth` labels of the data:

`[cat, cat, others, cat, others, others, cat, cat, cat, others]`

These class labels are currently in human-readable form; however, the model expects the ground truth to be numeric or integer. Therefore, when input is fed to the model, the model returns a score for each class. For example, the prediction of the model given the ten images from the dataset would be:

`[0.8, 0.1, 0.5, 0.9, 0.28, 0.35, 0.7, 0.4, 0.2, 0.68]`

Now we have the prediction from the model for each sample; for a binary classification problem, we can convert these prediction scores into class labels using a threshold. The threshold is a hyperparameter where you can experiment. For now, let’s assume the threshold to be `0.5`, which means if the model predicts above or equal to 0.5, then the sample belongs to a `cat` (positive) class; otherwise, it is `others` (negative) class.

After converting the prediction into class labels using the threshold, the predicted labels would look like this:

`prediction: [cat, others, cat, cat, others, others, cat, others, others, cat]`

If we compare both the ground-truth and predicted labels, we can see that there are five correct and five incorrect predictions made by the model at a threshold of `0.5`, and varying the threshold would alter the results.

Comparing the ground truth directly with prediction is fine, but what if we need a more in-depth analysis of our model’s performance?

For example, given a cat image, how many times does the model predict it as a cat or misclassify it as others? Similarly, for other class images, the number of times the model correctly predicted as others and misclassified as a cat.

Based on the above example, let’s define four new terms:

1.  **True Positive (TP):** The number of times the model correctly predicted the positive input sample (i.e., `cat`) as Positive.
2.  **False Positive (FP):** The number of times the model incorrectly predicted the negative sample (i.e., `others`) as Positive.
3.  **True Negative (TN):** The number of times the model correctly predicted the negative sample (i.e., `others`) as Negative.
4.  **False Negative (FN):** The number of times the model incorrectly predicted the positive input (i.e., `cat`) as Negative.

The above four terms help us extract deeper information about our model’s performance. Finally, let’s put them together to obtain Precision and Recall.

* * *

### **[Precision](#TOC)**

It is the ratio between the correctly classified Positive samples to the total number of samples classified as Positive (incorrect and correct). In short, Precision answers the question of how accurate the guesses were when the model guessed, or it measures the precision/accuracy of the model in classifying samples as positive.

In Precision, the focus is also on the Negative samples incorrectly classified as Positive since it has False Positive in its calculation. For example,

*   a patient with no sign of cancer is predicted as cancer or
*   a patch of an image detected as a person while there is no person in that patch

Precision can be calculated as

![Image 9: \text{Precision} = \displaystyle\frac{\text{TP}}{\text{TP}+\text{FP}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20153%2037'%3E%3C/svg%3E)

where ![Image 10: \text{TP}+\text{FP}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2065%2014'%3E%3C/svg%3E) are the total predicted objects.

For Precision to be high, the numerator needs to be higher (simple math). Precision would be high when the model correctly predicts the Positive samples as Positive (i.e., maximizing True Positive), and simultaneously, fewer (incorrect) Negative samples as Positive (i.e., minimizing False Positive).

The Precision would be low when the model predicts many Negative samples as Positive or fewer correct Positive classifications. This would increase the denominator and make the Precision smaller.

Now, let’s compute Precision in just a few lines of Python code.

import sklearn.metrics

ground\_truth = \["cat", "cat", "others", "cat", "others", "others","cat", "cat", "cat", "others" \]
prediction = \["cat", "others", "cat", "cat", "others", "others","cat", "others", "others", "cat"\]

precision = sklearn.metrics.precision\_score(ground\_truth, prediction, pos\_label="cat")
print(precision)

In the above lines of code, we use the `sklearn.metrics` method to compute Precision on the `cat` and `others` examples we took above to understand the binary classification problem.

It’s relatively straightforward, from **Lines 3 and 4**, we define `ground_truth` and `prediction` in a list. Then pass them to the `precision_score` method of `sklearn` on **Line 6**. We also give a `pos_label` argument that lets you specify which class should be considered as `positive` for the Precision computation.

* * *

### **[Recall](#TOC)**

It is the ratio between the correctly classified Positive samples to the total number of actual Positive samples. Recall answers whether your model guessed every time that it should be guessing. The higher the recall, the more positive samples are detected.

![Image 11: \text{Precision} = \displaystyle\frac{\text{TP}}{\text{TP}+\text{FN}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20154%2037'%3E%3C/svg%3E)

where ![Image 12: \text{TP}+\text{FN}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2066%2015'%3E%3C/svg%3E) are the total ground-truth objects.

As we learned, Precision does consider how the negative samples are classified; however, Recall is independent of how the negative examples are classified and only cares about how the positive samples are classified. This means that even if all the Negative samples are classified as Positive, the Recall would still be a perfect `1.0`.

The Recall would be `1.0` or 100% when all positive samples are classified as Positive. On the other hand, Recall would be low if Positive samples are classified as Negative; for example, imagine there are five cars in an image, and only four are detected, so False Negatives would be one.

import sklearn.metrics

ground\_truth = \["cat", "cat", "others", "cat", "others", "others","cat", "cat", "cat", "others"\]
prediction = \["cat", "others", "cat", "cat", "others", "others", "cat", "others", "others", "cat"\]

recall = sklearn.metrics.recall\_score(ground\_truth, prediction, pos\_label="cat")
print(recall)

Similar to Precision, we can calculate Recall by just changing the `sklearn.metrics.precision_score` to `sklearn.metrics.recall_score` on **Line 6**.

* * *

### **[TP, FP, TN, and FN in Detection Context](#TOC)**

We learned about Precision and Recall, and to calculate them, we need to compute True Positives, True Negatives, False Positives, and False Negatives. We also learned about Intersection over Union and how it helps evaluate an object detection model’s localization error. So, let’s know about Precision and Recall in object detection context and how **IoU** and **Confidence Score** play a role.

The **confidence score** reflects how likely the box contains an object (objectness score) and the bounding box’s accuracy. However, if no object exists in that cell, the confidence score should be zero.

* * *

#### **[True Positive](#TOC)**

A detection made by the model is considered True Positive only if it satisfies the two conditions:

*   The confidence score of the predicted bounding box should be greater than the confidence threshold (a hyperparameter), which would signify that we have found the object for which we were looking.
*   The IoU between the predicted bounding box and the ground-truth bounding box should be greater than the IoU threshold.

And what the above two conditions mean is that not only is the object detected, but it is detected at the correct location. **Figure 6** shows the True Positive detection with an IoU of `0.81` and a Confidence of `0.7`.

![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20287%20400'%3E%3C/svg%3E)

**Figure 6:** True Positive Detection.

* * *

#### **[False Positive](#TOC)**

A detection is considered to be a False Positive:

*   If the model detects an object with a high confidence score, but it is not present (no ground truth), then the IoU would be equal to zero.
*   Another scenario could be that the IoU is less than the IoU threshold.
*   If the proposed bounding box perfectly aligns with the ground truth, but the class label of the proposed box is incorrect.

![Image 14](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20416'%3E%3C/svg%3E)

**Figure 7:** Example depicting True Positive in green, False Positive in red, and Ground Truth in yellow ([source](https://www.cc.gatech.edu/classes/AY2016/cs4476_fall/results/proj5/html/pkundra3/index.html)).

* * *

#### **[True Negative](#TOC)**

This is generally not required, mainly because it is not part of the computation in Precision and Recall.

A True Negative in detection could be when the confidence score is less than the confidence threshold, and also the IoU (predicted box and ground-truth box) is less than the IoU threshold. In object detection, there could be a lot of True Negatives because the background covers more areas of the image than the object itself.

* * *

#### **[False Negative](#TOC)**

If there is an object present, but the model was not able to detect it, then that’s a False Negative.

![Image 15](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20287%20400'%3E%3C/svg%3E)

**Figure 8:** False Negative example with IoU less than threshold.

* * *

### **[Precision-Recall Curve](#TOC)**

Based on the learnings from the previous section, we now know that if the model has both high precision and recall, then it correctly predicts the samples as positive and predicts most of the positive samples (does not miss the samples or predict them as negative). However, if the model has high precision and low recall, it accurately predicts samples as positive but only a few positive samples (more false negatives).

Now that we know the importance of precision and recall and that the ultimate goal is to achieve the maximum scores for both precision and recall, let’s see how we can plot these on the same graph.

The plotting of the precision-recall values is known as a **precision-recall curve** that depicts the trade-off between the two based on different thresholds. The precision-recall curve helps to select the best threshold that maximizes both precision and recall.

Let’s understand the Precision-Recall curve with a binary classification example of cat and dog. In `utils_precision_recall.py`, we will write the `compute_precision_recall` method, which will take the ground-truth and prediction vectors along with 10 threshold values in a vector. Finally, return precision and recall for each of the 10 thresholds. It will also help plot the precision-recall curve.

\# import the necessary packages
from pyimagesearch import config
import matplotlib.pyplot as plt
import sklearn.metrics
import numpy as np

def compute\_precision\_recall(yTrue, predScores, thresholds):
    precisions = \[\]
    recalls = \[\]

    # loop over each threshold from 0.2 to 0.65
    for threshold in thresholds:
        # yPred is dog if prediction score greater than threshold
        # else cat if prediction score less than threshold
        yPred = \[
            "dog" if score >= threshold else "cat"
            for score in predScores
        \]
  
        # compute precision and recall for each threshold
        precision = sklearn.metrics.precision\_score(y\_true=yTrue,
            y\_pred=yPred, pos\_label="dog")
        recall = sklearn.metrics.recall\_score(y\_true=yTrue,
            y\_pred=yPred, pos\_label="dog")
  
        # append precision and recall for each threshold to
        # precisions and recalls list
        precisions.append(np.round(precision, 3))
        recalls.append(np.round(recall, 3))

    # return them to calling function
    return precisions, recalls

On **Line 7**, we define the `compute_precision_recall` function, which takes the ground-truth labels, predictions scores, and threshold list as an argument.

Then, we iterate through each threshold on **Line 12** convert the probabilities to a class label by checking if the prediction score (or probability) is greater than or equal to the threshold, we assign the `dog` label else `cat`.

From **Lines 21-24**, we compute the precision and recall for a given threshold and prediction. Then we round precision and recall to three decimal places and append them in a `precisions` and `recalls` list on **Lines 28 and 29**.

Finally, we return the `precisions` and `recalls` to the calling function on **Line 32**.

def pr\_compute():
    # define thresholds from 0.2 to 0.65 with step size of 0.05
    thresholds = np.arange(start=0.2, stop=0.7, step=0.05)

    # call the compute\_precision\_recall function
    precisions, recalls = compute\_precision\_recall(
        yTrue=config.GROUND\_TRUTH\_PR, predScores=config.PREDICTION\_PR,
        thresholds=thresholds,
    )
 
    # return the precisions and recalls
    return (precisions, recalls)

On **Line 36**, we define thresholds using `np.arange`, which creates a vector with values ranging from `0.2` to `0.65` with a step size of `0.05`.

From **Lines 39-42**, we call the `compute_precision_recall` method by passing in the `ground_truth`, `prediction`, and `thresholds` arrays.

def plot\_pr\_curve(precisions, recalls, path):
    # plots the precision recall values for each threshold
    # and save the graph to disk
    plt.plot(recalls, precisions, linewidth=4, color="red")
    plt.xlabel("Recall", fontsize=12, fontweight='bold')
    plt.ylabel("Precision", fontsize=12, fontweight='bold')
    plt.title("Precision-Recall Curve", fontsize=15, fontweight="bold")
    plt.savefig(path)
    plt.show()

![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20430'%3E%3C/svg%3E)

**Figure 9:** Precision-Recall curve plot.

In the above lines of code, we create a `plot_pr_curve` method which helps us plot the `precisions` and `recalls` list on a 2D graph with `precision` on the _x_\-axis and `recall` on the _y_\-axis. Notice how the precision starts to drop as recall increases after 0.6.

**We have our precision-recall curve, but how do we know at which threshold the model performs the best (i.e., high precision and recall)?**

To find out the best precision and recall, we use F1-score to combine precision and recall into a single metric by taking their harmonic mean. A higher F1-score would mean that precision and recall are high, while a lower F1-score signifies a high imbalance between precision and recall (or lower precision and recall).

![Image 17: \text{F1} = \displaystyle\frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \displaystyle\frac{2\times\text{TP}}{2\times(\text{TP}+\text{FP}+\text{FN})}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20373%2040'%3E%3C/svg%3E)

f1\_score = np.divide(2 \* (np.array(precisions) \* np.array(recalls)), (np.array(precisions) + np.array(recalls)))

print(f1\_score)

array(\[0.71959027, 0.69536849, 0.69536849, 0.69986987, 0.73694181, 0.70606916, 0.70606916, 0.66687977, 0.57106109, 0.46121884\])

On **Line 1**, we compute the `f1_score` by passing the `precisions` and `recalls` list we got from above. From the output, we can observe that the highest `f1_score` is `0.7369` at index 5.

Now, let’s use the highest `f1_score` to get the best threshold that balances precision and recall.

precision = precisions\[np.argmax(f1\_score)\]
recall = recalls\[np.argmax(f1\_score)\]
best\_threshold = np.round(thresholds\[np.argmax(f1\_score)\], 1)

print(precision, recall, best\_threshold)

(0.7, 0.778, 0.4)

From **Lines 7-9**, we compute the precision, recall, and best threshold by passing the `f1_score` to the `np.argmax` function to get the maximum value index in the `f1_score` array.

Finally, we can conclude that the best threshold to balance precision and recall is `0.4`. Another way to put this is to expect our model to achieve optimal precision and recall at the `0.4` threshold.

* * *

### **[Object Detection Competition Datasets](#TOC)**

* * *

#### **[PASCAL VOC 2012](#TOC)**

The [PASCAL Visual Object Classes 2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html) (VOC2012) is a challenge to recognize objects from various visual object classes in a realistic scenario. It is a supervised learning challenge in which the labeled ground-truth images are provided. The dataset has 20 object classes like a person, bird, cat, dog, bicycle, car, chair, sofa, tv, bottle, etc.

The history of PASCAL VOC dates back to the year 2005 when the dataset consisted of only four classes: bicycles, cars, motorbikes, and people. It had a total of 1578 images containing 2209 annotated objects.

**Figure 10** shows the sample images from each of the 20 classes, and as we can see, there are several (three) chairs in the second last image in the first row.

![Image 18](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201024%20201'%3E%3C/svg%3E)

**Figure 10:** Example images from PASCAL VOC dataset depicting all 20 classes ([source](https://cocodataset.org/#home)).

The VOC2012 dataset consists of 11,530 images with 27,450 Region of Interest (ROI) annotated objects with a train/val split. The 27,450 ROI refers to the bounding boxes in the entire dataset since each image can have multiple objects or ROI. Hence, there are more than 2X ROIs compared to images.

* * *

#### **[MS COCO](#TOC)**

The [Microsoft Common Objects in Context](https://cocodataset.org/#home) (MS COCO) is large-scale object detection, segmentation, and captioning dataset introduced in the year 2014 with the help of extensive crowdsourcing using novel user interfaces. The dataset contains images of 80 object categories with 2.5 million labeled instances in 328k images.

**Figure 11** shows the example images from the COCO dataset labeled with instance segmentation.

![Image 19](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20213'%3E%3C/svg%3E)

**Figure 11:** COCO Dataset Sample Images ([source](https://cocodataset.org/#home)).

COCO also has an excellent user interface to explore the images within the dataset. For example, you can select the thumbnails from 80 classes; it will put them in the search bar as tags, and when you search, it will show all the images from the dataset with those tags (classes), as shown in **Figure 12**.

![Image 20](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20473%20500'%3E%3C/svg%3E)

**Figure 12:** COCO Explorer ([source](https://cocodataset.org/#explore)).

Further, all the classes (more than the searched classes) are displayed as thumbnails when the resulting images are shown. You can play around with it by further clicking on those thumbnails to visualize segmentation masks related to those classes.

* * *

### **[Precision-Recall Curve for Object Detection](#TOC)**

The Precision-Recall curve holds the same or even more importance in evaluating an object detection model’s performance compared to classification. An object detector is considered good if the precision and recall are not impacted when we vary the confidence threshold (probability that a box contains an object). Also, the precision does not decrease as the recall increases.

An ideal object detector with high precision and high recall would have zero false positives (only detect relevant objects) and zero false negatives (not missing relevant objects or predicting all ground-truth objects).

Let’s now try to understand how Precision and Recall curve with the help of an example. _For this and subsequent examples, we would use Rafael Padilla’s object detection metrics_ [_GitHub repository_](https://github.com/rafaelpadilla/Object-Detection-Metrics) _that provides an excellent summary with examples on precision-recall and average precision._

Consider an example shown in **Figure 13**; seven images consist of 15 ROIs or ground-truth bounding boxes (shown in green) and 24 ROI proposals or predicted bounding boxes (shown in red). Each of the predicted bounding boxes has a confidence score associated with it.

![Image 21](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20392'%3E%3C/svg%3E)

**Figure 13:** Seven images consisting of 15 ground-truth bounding boxes in green and 24 proposed ROIs in red ([source](https://github.com/rafaelpadilla/Object-Detection-Metrics)).

Now let’s transfer all these seven images data into **Table 1**, which has four columns: image names, detections, a confidence score for each detection, and True Positive (TP) or False Positive (FP). For this example, assume the ![Image 22: \text{IoU} \geq 0.5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2069%2016'%3E%3C/svg%3E).

![Image 23](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20235%20500'%3E%3C/svg%3E)

**Table 1:** Proposed ROIs with their respective confidences and TP/FP ([source](https://github.com/rafaelpadilla/Object-Detection-Metrics)).

For `Image 1`, we have three detections: `A`, `B`, and `C` with their respective confidence scores `88%`, `70%`, and `80%` and whether these detections are TP or FP. If we observe `Image 1`, we can see that the detection `A` is an FP. This is because it has significantly less overlap with the GT, even though the confidence is high. Remember, we discussed TP/FP in the context of object detection. The detection `B` seems to be aligned quite well with the GT, so it is marked as a TP, while the detection `C` is again an FP as there is no GT present for the proposal. And accordingly, we have the TP and FP for the rest of the six images.

We get all TP/FP for the dataset (seven images), but we need to compute TP/FP accumulated over the entire dataset to arrive at the Precision-Recall curve. Therefore, we first arrange the above table rows based on the highest confidence score to the lowest, with which we get a new **Table 2**.

**Table 2** has five more columns: TP/FP are split into two separate columns, accumulated TP, accumulated FP, Precision, and Recall. To calculate Precision and Recall, we use the same formula discussed previously.

![Image 24](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20417%20500'%3E%3C/svg%3E)

**Table 2:** Proposed ROIs with their respective accumulated TP/FP and Precision/Recall over all the ROIs ([source](https://github.com/rafaelpadilla/Object-Detection-Metrics)).

Let’s break down **Table 2** into the following steps:

1.  We segregate the TP and FP into two columns, and for each proposed ROI, we specify 1 and 0 in their respective columns. For example, in `Image 5`, detection `R` is a TP, so we specify 1 in the TP column for the first row, while, in the same image, detection `Q` is an FP, so we set 1 in the FP column in the 18th row. The same idea is followed for all the proposed ROIs.
2.  Once we are through with TP and FP for all the proposed ROIs, we move to the following two columns: `Acc TP` and `Acc FP`. Calculating them is pretty simple. For the first row, `Acc TP` would be the same as TP (i.e., 1). For the second row, it would remain the same as TP is 0. For the third row, it would be 2 as the TP is 1 (so far, two TPs: 1st and 3rd rows). Similarly, we would calculate `Acc TP` for all the rows until we reach the end.
3.  Then, we will follow the same exercise for the `Acc FP` column and fill all the rows based on the values from the previous `Acc FP` and current row FP.
4.  Now that we have the `Acc TP` and `Acc FP`, it is straightforward to calculate the Precision and Recall. For each proposed ROI, we would calculate Precision using the `Acc TP` and `Acc FP`, and compute Recall with `Acc TP` as the numerator and the number of GT as the denominator (TP+FN).
5.  For the first row, the Precision will be ![Image 25: \displaystyle\frac{1}{1+0}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2037%2037'%3E%3C/svg%3E), which is nothing but `1`; for the second row, it will be ![Image 26: \displaystyle\frac{1}{1+1}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2037%2037'%3E%3C/svg%3E), which is `0.5`, and we will calculate it for all the rows. Similarly, we can calculate the Recall; for example, for the last row, it will be ![Image 27: \displaystyle\frac{7}{15}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2017%2037'%3E%3C/svg%3E) (i.e., `0.466`).

Great, now that we have precision and recall values for all the proposed ROIs, we can plot a graph of Precision vs. Recall, as shown in **Figure 14**.

![Image 28](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20566%20422'%3E%3C/svg%3E)

**Figure 14:** Precision-Recall curve for all the proposed ROIs (inspired from [source](https://github.com/rafaelpadilla/Object-Detection-Metrics)).

As we can see from the above figure, we have our Precision-Recall curve that shows all the proposed ROIs with the highest confidence score `R` at the top-left shown in green and the detection with the lowest confidence score displayed at the bottom-right in red color.

Now that we have the Precision-Recall (PR) curve, the next step is to calculate the area under the PR curve (i.e., Average Precision (AP)). However, before calculating the AP, we need to smooth out the zig-zag pattern (i.e., the noise from the above graph) since the precision and recall keep going up and down.

* * *

### **[Calculating the Average Precision](#TOC)**

As learned above, Average Precision (AP) finds the area under the precision-recall curve; we can compute the Average Precision from the PR curve using the 11-point interpolation technique introduced in the PASCAL VOC challenge.

Let’s see how we can apply this technique to the PR curve and arrive at the Average Precision.

* * *

#### **[11-Point Interpolation](#TOC)**

The 11-point interpolation calculates the Precision at the Recall levels of ![Image 29: [0, 0.1, 0.2, \dots, 1]](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20115%2018'%3E%3C/svg%3E) (i.e., 11 equally spaced recall levels), and then averages them out. Mathematically it can be expressed as:

![Image 30: \text{AP}=\displaystyle\frac{1}{11} \displaystyle\sum_{r\in \left \{ 0, 0.1, \dots,1 \right \}}\rho_{\text{interp}\left (r\right)}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20195%2047'%3E%3C/svg%3E)

![Image 31: \rho_\text{interp} =  \max\limits_{\bar{r}:\bar{r} \geq r} \rho\left(\tilde{r}\right)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20125%2027'%3E%3C/svg%3E)

where ![Image 32: \rho\left(\tilde r\right)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2030%2018'%3E%3C/svg%3E) is the measured precision at recall ![Image 33: r](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%207'%3E%3C/svg%3E).

Since Precision and Recall are between `0` and `1`, the AP also falls between `0` and `1`. Therefore, before calculating the AP, we need to smooth out the zig-zag pattern from the PR curve.

![Image 34](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20521%20417'%3E%3C/svg%3E)

**Figure 15:** Smoothed-out Precision-Recall curve in red (inspired from [source](https://github.com/rafaelpadilla/Object-Detection-Metrics)).

At each recall level, we replace each precision value with the maximum precision value to the right of that recall level, as shown in **Figure 16**.

![Image 35](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20521%20417'%3E%3C/svg%3E)

**Figure 16:** Smoothed-out curve is obtained by moving each precision value with the maximum precision value to the right of that recall level (inspired from [source](https://github.com/rafaelpadilla/Object-Detection-Metrics)).

So the blue line is transformed into the red lines, and the curve will decrease monotonically instead of the zig-zag pattern. As a result, the calculated AP value will be less susceptible to slight variations in the ranking. Mathematically, we replace the precision value for recall _ȓ_ with the maximum precision for any recall ≥ _ȓ_.

Once we have this graph, we can divide the recall values from `0` to `1.0` into 11 equal intervals of 0.1 to arrive at **Figure 17**.

Let’s now compute the AP based on the below 11-point interpolated graph:

![Image 36: \text{AP} = \displaystyle\frac{1}{11}(1 + 0.66 + 3 \times 0.4285 + 6 \times 0)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20283%2036'%3E%3C/svg%3E)

![Image 37: \text{AP} = \displaystyle\frac{1}{11}(2.9521)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20121%2036'%3E%3C/svg%3E)

![Image 38: \text{AP} = 0.2683](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2090%2013'%3E%3C/svg%3E)

![Image 39: \text{AP} = {26.83}{\%}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2095%2015'%3E%3C/svg%3E)

![Image 40](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20678%20500'%3E%3C/svg%3E)

**Figure 17:** Precision at 11 recall levels (inspired from [source](https://github.com/rafaelpadilla/Object-Detection-Metrics)).

**_Note:_** _According to the original researcher, the intention of interpolating the precision/recall curve is to reduce the impact of the “wiggles” in the precision/recall curve caused by slight variations in the ranking of examples._

* * *

### **[Mean Average Precision (mAP)](#TOC)**

We can calculate the mAP by simply taking the mean over all the class APs. For example, in the PASCAL VOC dataset, we can compute an AP for each of the 20 categories and then average over all the 20 AP classes to get the mean average precision.

**Table 3** shows the mAP of various detectors (e.g., SSD300 and SSD512) on the PASCAL VOC dataset and AP of each of the 20 classes. The AP over each class gives a more fine-grained evaluation of the object detector since it tells about the classes where the detector did well and the classes in which the detector performed poorly.

![Image 41](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20177'%3E%3C/svg%3E)

**Table 3:** mAP of various detectors like SSD300 and SSD512 on the PASCAL VOC dataset and AP of each of the 20 classes ([Liu et al.](https://arxiv.org/pdf/1512.02325.pdf), p. 8, table 1).

* * *

**[Evaluating the YOLOv4 Model Using the COCO Evaluator](#TOC)**
----------------------------------------------------------------

This final section will learn to evaluate the object detection model’s performance using the COCO evaluator. We will use the [YOLOv4](https://arxiv.org/pdf/2004.10934.pdf) object detector trained on the MS COCO dataset, and it achieved state-of-the-art results: 43.5% AP (65.7% AP50) for the MS COCO dataset at a real-time speed of `∼65` FPS on the Tesla Volta100 GPU.

We would evaluate the YOLOv4 model on the 2017 MS COCO Validation dataset with 5000 images of varying sizes with the ground truth in COCO format.

* * *

### **[COCO mAP Evaluator](#TOC)**

The COCO evaluator is now the gold standard for computing the mAP of an object detector. Most of the research papers provide benchmarks for the COCO dataset using the COCO evaluation from the past few years. And there are two main reasons. First, the dataset is much richer than the VOC dataset. Secondly, the COCO evaluation does a much deeper analysis of the detection model, thereby providing complete results.

As learned in PASCAL VOC evaluation, a proposed ROI is considered True Positive if ![Image 42: \text{IoU} \geq 0.5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2069%2016'%3E%3C/svg%3E) with the ground truth. Since there is only one threshold, hence, a detection that very closely aligns (![Image 43: \text{IoU} = 0.95](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2077%2014'%3E%3C/svg%3E)) with the ground truth would be considered similar to the one with ![Image 44: \text{IoU} = 0.5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2069%2014'%3E%3C/svg%3E).

Having a single IoU threshold to assess our detection model might not be a good idea since a single threshold can induce a bias in the evaluation metric. Also, it can be lenient for the model. Therefore, the COCO mAP evaluator averages the mAP of 80 classes over 10 IoU thresholds from 0.5 to 0.95 with a step size of 0.05 (AP@\[0.5:0.05:0.95\]).

It is worth noting that, unlike PASCAL VOC, the COCO evaluator uses 101-point interpolated AP (i.e., it calculates the precision values at 101 recall levels \[0:0.01:1\]).

**Figure 18** shows the variation of object detection evaluation that the COCO evaluator provides us with, which we will also learn to calculate in the next section of the tutorial.

![Image 45](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201024%20460'%3E%3C/svg%3E)

**Figure 18:** The object detection COCO evaluation metric at various IoUs, Scales ([source](https://cocodataset.org/#detection-eval)).

The above figure shows that the COCO evaluator can report AP in various ways, that is, AP@\[0.5:0.05:0.95\], AP@0.5, AP@0.75, AP across three scales (small, medium, and large).

AP across three scales can be very beneficial if your use case requires you to detect a lot of small objects, then you would want to maximize ![Image 46: \text{AP}^\text{small}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2050%2016'%3E%3C/svg%3E).

**Figure 19** shows the COCO format of the ground truth in the JSON format. It has an optional `info` field with some information about the year of data publication, version, and URL related to the dataset.

Then, the `image` field contains the image’s metadata like `width`, `height`, `file_name`, etc.

![Image 47](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20625%20500'%3E%3C/svg%3E)

**Figure 19:** COCO format of the ground-truth in the JSON format ([source](https://cocodataset.org/#format-data)).

Since the COCO dataset is not just for object detection tasks but also for segmentation, image captioning, and keypoint detection, the annotations would differ for each task. Let’s see what the `annotations` format looks like for object detection.

**Figure 20** shows the object detection `annotation` and `categories` fields. The `annotation` field has `image_id`, which would be the same for all ROIs, `category_id` that is the class id, `area` would be the area of the bounding box (![Image 48: \text{width} \times \text{height}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20104%2016'%3E%3C/svg%3E)), `bbox` – top-left corner coordinates, width, and height of the bounding box.

Finally, the `categories` field will have class data (i.e., class id, class name (car, person)) and supercategory. So, for example, a car and bicycle would have the same supercategory (i.e., vehicle).

![Image 49](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20275'%3E%3C/svg%3E)

**Figure 20:** Object detection annotation and categories fields in the ground truth ([source](https://cocodataset.org/#format-data)).

But we do not have to convert our dataset ground-truth to the above COCO format since the COCO Val2017 dataset is already in the desired format.

This was about the ground-truth format, but let’s see how the prediction format in COCO looks.

As we can see in **Figure 21**, the prediction format is quite simple; we have the `image_id`: image name, `category_id`: class id, `bbox`: top-left coordinates, width and height of bounding box, and `score`: confidence score of predictions.

![Image 50](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20125'%3E%3C/svg%3E)

**Figure 21:** Object detection result format ([source](https://cocodataset.org/#format-results)).

Now that we fully understand the ground-truth and prediction COCO format, let’s get straight into the code and compute the mean average precision of the YOLOv4 model.

_**Note:**_ We will use YOLOv4 from [AlexeyAB’s Darknet Repository](https://github.com/AlexeyAB/darknet), and configuring it in Google Colab is a cakewalk.

* * *

### **[Configuring the Prerequisites](#TOC)**

For today’s task, we will use the **MS COCO Val 2017** dataset. We created this dataset for object detection, segmentation, and image captioning purposes. It contains 5000 images in the validation set.

Before proceeding to the code, be sure you have configured the Darknet framework and downloaded the YOLOv4 weights and the MSCOCO Val 2017 images and labels.

If you are using a non-Linux system, you can get the dataset [here](https://cocodataset.org/#download). This will initiate the zipped dataset’s download. Then, extract the contents of this zip file to the project’s `data` directory.

If you are using a Linux system, you can simply follow the instructions below:

$ cd data
$ wget http://images.cocodataset.org/zips/val2017.zip
$ unzip val2017.zip
$ wget -c http://images.cocodataset.org/annotations/annotations\_trainval2017.zip
$ unzip annotations\_trainval2017.zip
$ wget https://github.com/AlexeyAB/darknet/releases/download/yolov4/yolov4.weights
$ cd ../

On **Line 1**, we first change the directory to `data`, then on **Lines 2 and 3**, we download and unzip the `val2017` images. Then, we download and unzip the ground-truth annotations on **Lines 4 and 5**. Finally, on **Line 6**, we download the `YOLOv4` weights file.

To compute mean average precision, we open the `utils_map.py` script, which has a few very important methods for detecting objects: converting bounding boxes to the required format, loading pickle files, running inference over the image directory, and finally storing them prediction in JSON format.

\# import the necessary packages
from pyimagesearch import config
from progressbar import progressbar
from darknet import darknet
import pickle
import json
import cv2
import os

def image\_detection(imagePath, network, classNames, thresh):
	# image\_detection takes the image as an input and returns
	# the detections to the calling function
	width = darknet.network\_width(network)
	height = darknet.network\_height(network)
	
	# create an empty darknetImage of shape \[608, 608, 3\]
	darknetImage = darknet.make\_image(width, height, 3)

	# read the image from imagePath, convert from BGR->RGB
	# resize the image as per YOLOv4 network input size
	image = cv2.imread(imagePath)
	imageRGB = cv2.cvtColor(image, cv2.COLOR\_BGR2RGB)
	imageResized = cv2.resize(imageRGB, (width, height),
		interpolation=cv2.INTER\_LINEAR)

	darknet.copy\_image\_from\_bytes(darknetImage, imageResized.tobytes())
	
	# detections include the class labels, bounding box coordinates, and
	# confidence score for all the proposed ROIs in the image
	detections = darknet.detect\_image(network, classNames, darknetImage,
		thresh=thresh)
	darknet.free\_image(darknetImage)
	
	# return the detections and image shape to the calling function
	return (detections, image.shape)

The first method we define is `image_detection`. As the name suggests, it takes the image as an input and returns the detections to the calling function (**Lines 10-35**).

The detections include the class labels, bounding box coordinates, and confidence score for all the proposed ROIs in the image. We also return the raw image shape since we would need this later to scale the predicted bounding box coordinates.

From **Lines 13-17**, we define the network input width and height which it gets from the `yolov4.cfg` file and create an empty `darknetImage` of shape `[608, 608, 3]`.

Then from **Line 21-24**, we read the image from the path, convert the image to RGB color space since OpenCV reads in BGR format and resize the image as per the network width and height.

Next, we copy the contents of the image to the empty `darknetImage` and finally call the `detect_image` method with a few parameters like the `network`, `classNames`, `darknetImage`, and `thresh`. It then returns `detections` with proposed bounding box coordinates, class labels, and a confidence score for each proposed ROI.

Finally, we return the `detections` and `image.shape` to the calling function on **Line 35**.

def bbox2points(bbox, img\_shape):
    #from bounding box yolo format centerx, centery, w, h
    #to corner points top-left and bottom right
    (x, y, w, h) = bbox

    # compute scale ratio dividing by 608
    hRatio = img\_shape\[0\] / config.YOLO\_NETWORK\_HEIGHT
    wRatio = img\_shape\[1\] / config.YOLO\_NETWORK\_WIDTH
    xMin = int(round(x - (w / 2)))
    yMin = int(round(y - (h / 2)))

    # scale them as per the original image size
    return (xMin \* wRatio, yMin \* hRatio, w \* wRatio, h \* hRatio)

From **Lines 37-49**, we define the `bbox2points` method that accepts `bbox` coordinates and `img_shape`: the shape of the input image fed to the model.

The YOLOv4 model outputs the bounding box predictions in ![Image 51: [c_{x}, c_{y}, w, h]](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2078%2018'%3E%3C/svg%3E) format where ![Image 52: c_{x}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2014%2010'%3E%3C/svg%3E) and ![Image 53: c_{y}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2012'%3E%3C/svg%3E) are the center coordinates of the bounding box, ![Image 54: w](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%207'%3E%3C/svg%3E) and ![Image 55: h](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2011'%3E%3C/svg%3E) are the width and height of the bounding box. These coordinates are normalized per network input size (i.e., `608)` since we use the YOLOv4 model with `608` input resolution.

However, we need to convert the bounding box predictions into ![Image 56: [\text{left}_{x}, \text{top}_{y}, w, h]](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20110%2019'%3E%3C/svg%3E) format and also scale them as per the original image size. We do this so that the `bbox` prediction format matches the ground-truth `bbox` format.

We compute the scale ratio on **Lines 43 and 44** by dividing the original image size by the network input size ![Image 57: (608 \times 608)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2079%2018'%3E%3C/svg%3E). Then we calculate the top-left coordinates: `xmin` and `ymin` and finally return the scaled top-left, width, and height of the bounding box to the calling function.

def load\_yolo\_cls\_idx(path):
    # load pickle files: COCO 'class\_id' to the
    # class\_name list and the COCO 0-90 class ids list
    with open(path, "rb") as f:
        out = pickle.load(f)
        return out

The `load_yolo_cls_idx` function is a utility function that loads the COCO `class_id` to the `class_name` list and the COCO 0-90 class ids list. We need to map the COCO 0-80 class ids to 0-90 class ids since the COCO Val2017 ground-truth `category_id` ranges from 0 to 90.

def run\_inference(imagePaths, network, class\_names,
    label2idx, yolo\_cls\_90, conf\_thresh, pred\_file):
    results = \[\]

    # loop over all the images
    for path in progressbar(imagePaths):
        # pass the imagePath, network, class names and confidence
        # threshold to image\_detection method that returns detections
        # and original image shape
        detections, imgShape = image\_detection(
            path, network, class\_names, thresh=conf\_thresh
        )

       # extract imageID
       imageId = int(os.path.splitext(os.path.basename(path))\[0\])

       # loop over all the proposed ROIs
       for cls, conf, box in detections:
           (x, y, w, h) = bbox2points(box, imgShape)
           label = \[key for key, value in label2idx.items()
                   if value == cls\]\[0\]
           label = yolo\_cls\_90\[label\]

           # append result for each ROI as per COCO prediction
           # in a dictionary format
           results.append({"image\_id": imageId,
                           "category\_id": label,
                           "bbox": \[x, y, w, h\],
                           "score": float(conf)})

           # save the results on disk in a JSON format
           with open(pred\_file, "w") as f:
               json.dump(results, f, indent=4)

The `run_inference` method is where most of the magic happens:

*   Iterating over the images to calling `image_detection` for detections
*   Get the `imageId` from the tail of the path
*   Looping over proposed detections
*   Transforming the proposed bounding box coordinates with the `bbox2points` method
*   Then extracting label id and converting them to 0-90 class index
*   Appending all the required information in a `results` list
*   Finally, creating and dumping the results in a JSON format

Now that most of the work is done in our previous script, we define the `eval_map.py` script, which is the main driver script that loads the `YOLOv4` model from the disk and calls the required methods in the `utils_map.py` script. Finally, it uses the `COCOeval` APIs to compute the mAP of the `YOLOv4` detector on the COCO Val 2017 dataset.

\# import the necessary packages
from pyimagesearch.utils\_map import run\_inference
from pyimagesearch.utils\_map import load\_yolo\_cls\_idx
from pyimagesearch import config
from pycocotools.cocoeval import COCOeval
from pycocotools.coco import COCO
from darknet import darknet

def compute\_map():
	# use the COCO class to load and read the ground-truth annotations
	cocoAnnotation = COCO(annotation\_file=config.COCO\_GT\_ANNOTATION)
	
	# call the darknet.load\_network method, which loads the YOLOv4
	# network based on the configuration, weights, and data file
	(network, classNames, \_) = darknet.load\_network(
			config.YOLO\_CONFIG,
			config.COCO\_DATA,
			config.YOLO\_WEIGHTS,
		)

	label2Idx = load\_yolo\_cls\_idx(config.LABEL2IDX)
	yoloCls90 = load\_yolo\_cls\_idx(config.YOLO\_90CLASS\_MAP)

	# call the run\_inference function to generate prediction JSON file
	run\_inference(config.IMAGES\_PATH, network, classNames, label2Idx,
		yoloCls90, config.CONF\_THRESHOLD, config.COCO\_VAL\_PRED)	

On **Line 11**, we use the `COCO` class to load and read the ground-truth annotations.

From **Lines 15-19**, we call the `darknet.load_network` method, which loads the YOLOv4 network based on the configuration, weights, and data file. For example, if the configuration file has width and height set to 512, it will create the model accordingly. It also returns us the class names.

We then load the `label2Idx` dictionary and `yoloCls90` map list on **Lines 21 and 22**.

Now that we have the `image_paths` list defined, a network created, class names, and other helper files loaded, we can call the `run_inference` function on **Lines 25 and 26** by passing these as parameters. Once the function execution is complete, a `COCO_Val_Prediction.json` file is created, and the sample prediction JSON is shown in **Figure 22**.

![Image 58](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20448%20500'%3E%3C/svg%3E)

**Figure 22:** Sample object detection result JSON file.

	# load detection JSON file from the disk
	cocovalPrediction = cocoAnnotation.loadRes(config.COCO\_VAL\_PRED)

	# initialize the COCOeval object by passing the coco object with
	# ground truth annotations, coco object with detection results
	cocoEval = COCOeval(cocoAnnotation, cocovalPrediction, "bbox")
	
	# run evaluation for each image, accumulates per image results
	# display the summary metrics of the evaluation
	cocoEval.evaluate()
	cocoEval.accumulate()
	cocoEval.summarize()

On **Line 29**, we load the prediction annotation file that is generated by the `run_inference()` method.

Then, on **Line 33**, we initialize the `COCOeval` object by passing the coco object with ground-truth annotations (`instances_val2017.json`) and the coco object with detection results (`COCO_Val_Predictions.json`). The third parameter is interesting; it is the `IoUType` which can be `segm` for segmentation evaluation, `keypoints` for keypoint detection evaluation. However, we need to evaluate an object detection model, so we pass `bbox`.

Finally, from **Lines 37-39**, we call three functions sequentially:

*   `cocoEval.evaluate` which runs evaluation for each image
*   `cocoEval.accumulate` that accumulates per image results
*   `cocoEval.summarize` that displays the summary metrics of the evaluation

Finally comes the moment of truth! The YOLOv4 model achieved 65.8mAP@0.5 IoU and 45.8mAP@\[0.5:0.95\] Average IoU on the COCO Val 2017 dataset (**Figure 23)**. It took a total of 267 seconds for the `run_inference` method to complete, which means the YOLOv4 model achieved more than 19 FPS on a Tesla T4; however, this number could vary on Google Colab.

![Image 59](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201024%20687'%3E%3C/svg%3E)

**Figure 23:** YOLOv4 mAP on the COCO Val 2017 dataset.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

**[Summary](#TOC)**
-------------------

Congratulations on making it this far! We hope you learned a lot through this tutorial, so let’s summarize what we learned quickly.

1.  We started by discussing the concept of IoU, why we use IoU, and learned to compute IoU in Python.
2.  Then we discussed Precision and Recall from a classification perspective, which helped us understand True Positives, False Positives, and False Negatives from an object detection context with examples.
3.  Next, we discussed two of the most popular object detection datasets, PASCAL VOC and MS COCO, which also devised ways to compute mAP with 11-point and 101-point interpolation.
4.  We then discussed the Precision-Recall curve for object detection and learned to calculate the Average Precision using the 11-point interpolation technique with the help of an example.
5.  Finally, we learned how to evaluate the YOLOv4 model on the MS COCO Val2017 dataset with a COCO evaluator in Python.

* * *

### **[Citation Information](#TOC)**

**Sharma, A.** “Mean Average Precision (mAP) Using the COCO Evaluator,” _PyImageSearch_, D. Chakraborty, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, R. Raha, and A. Thanki, eds., 2022, [https://pyimg.co/nwoka](https://pyimg.co/nwoka)

@incollection{Sharma\_2022\_mAP,
  author = {Aditya Sharma},
  title = {Mean Average Precision {(mAP)} Using the {COCO} Evaluator},
  booktitle = {PyImageSearch},
  editor = {Devjyoti Chakraborty and Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/nwoka},
}

* * *

![Image 60: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 61](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!