Pipeline for comparing two object detection models:
By: Naveen Yadav
Tags: Machine Learning, Precision, Object Detection, Average, Deep Learning

Before proceeding further, there are some terms which are needed to be clear regarding object detection. Those terms are mentioned below:

IoU: Intersection over union is a ratio to calculate how much area of the detection covers the actual ground truth box.

Formula IoU: Detected Bounding Box ⋂ Ground Truth Box

----------------------------------------

Detected Bounding Box ⋃ Ground Truth Box

True Positives: Number of correct detections which have IoU_with_ground_truth_box > IoU_threshold. False Positives: Number of incorrect detections which have IoU_with_ground_truth_box < IoU_threshold. False Negative: When object is not detected then that object is counted as false negative. That particular object is considered as background. True Negative: Background.(Which is not important) Precision: True positive rate of the model. This implies that how correctly the model is detecting objects in a image.

Precision = TP

------

TP + FP

Recall: This implies how often the model detects which should be detected in an image.

Recall = TP

--------

TP + FN

Precision vs Recall curve:

There’s a popular theory that if we want to increase model’s precision i.e to make sure that model only predicts objects with higher confidence then it may reduce the recall value , this is called precision recall tradeoff.

For example, consider that there are four dogs in an image and the model detects only one dog in the image so model has high precision but has poor recall because it only detected one dog out of four .

. So as we decrease the confidence threshold , model’s recall value increase gradually by decreasing the precision.

Note: Different colors shows the tradeoff between different classes.

Mean Average Precision:

So we consider precision at every point decrease in confidence_threshold as shown in the graph .ie. we gradually decrease the confidence_threshold and consider precision for that particular point.

Take weighted mean of all precision points where as the increase in the recall value from the previous point are considered as the weights.

AP = summation of (Wi x Pi)

------------------------

Number_of_(Pi) Wi = increase in recall from previous Precision Point. Pi = current precision point

Now we take plots of precision vs recall at different iou_thresholds by gradually decreasing iou_thresholds for each different class.

Pipeline for comparing two object detection models:

Before proceeding further, there are some terms which are needed to be clear regarding object detection. Those terms are mentioned below:

IoU: Intersection over union is a ratio to calculate how much area of the detection covers the actual ground truth box.

Formula IoU: Detected Bounding Box ⋂ Ground Truth Box

----------------------------------------

Detected Bounding Box ⋃ Ground Truth Box

True Positives: Number of correct detections which have IoU_with_ground_truth_box > IoU_threshold. False Positives: Number of incorrect detections which have IoU_with_ground_truth_box < IoU_threshold. False Negative: When object is not detected then that object is counted as false negative. That particular object is considered as background. True Negative: Background.(Which is not important) Precision: True positive rate of the model. This implies that how correctly the model is detecting objects in a image.

Precision = TP

------

TP + FP

Recall: This implies how often the model detects which should be detected in an image.

Recall = TP

--------

TP + FN

Precision vs Recall curve:

There’s a popular theory that if we want to increase model’s precision i.e to make sure that model only predicts objects with higher confidence then it may reduce the recall value , this is called precision recall tradeoff.

For example, consider that there are four dogs in an image and the model detects only one dog in the image so model has high precision but has poor recall because it only detected one dog out of four .

. So as we decrease the confidence threshold , model’s recall value increase gradually by decreasing the precision.

Note: Different colors shows the tradeoff between different classes.

Mean Average Precision:

So we consider precision at every point decrease in confidence_threshold as shown in the graph .ie. we gradually decrease the confidence_threshold and consider precision for that particular point.

Take weighted mean of all precision points where as the increase in the recall value from the previous point are considered as the weights.

AP = summation of (Wi x Pi)

------------------------

Number_of_(Pi) Wi = increase in recall from previous Precision Point. Pi = current precision point

Now we take plots of precision vs recall at different iou_thresholds by gradually decreasing iou_thresholds for each different class.

Average all precisions at different points of iou_threshold

We again mean the values over all class mAPs.