A Comprehensive Guide To YOLOv3!!!
By: Atharva Musale
Tags: Object Detection, Yolov3, Deep Learning

Table of Contents-

Introduction Problem Definition Performance Metric Tradeoff Between Speed vs mAP Darknet-53 Bounding Box and Output Per class Sigmoid Loss Function Multi-Scale Prediction Combining boxes from various scale- Total working Conclusion References

1. Introduction —

YOLOv3 is an object detection algorithm which is one the most widely used algorithm in the field of Computer Vision. In this blog, I am going to discuss the theoretical aspects of the YOLOv3 and in the next few blogs, I will be writing about the implementation details of YOLOv3 for object detection and also about object tracking using Deep Sort.

2. Problem Definition —

Object detection Output

In the above image, we have many objects and the bounding boxes around those. Our task is to predict the bounding boxes around the desired object classes using the YOLOv3 algorithm. As you can see this task is not as rigorous as the segmentation task where segmentation works on pixel-level depth. Object detection task is computationally much simpler as compared to segmentation task and that's the reason why it's used in day to day applications like self-driving cars.

3. Performance Metric —

It is very important to know the business metric before attempting any Deep Learning Problem. Here are 3 ways by which you can measure the performance of a YOLOv3.

IOU score is the metric that we generally use for Object Detection. We choose some threshold on the IOU score to classify the detection as correct or not. But this metric can only be used for the images which are having a single object in an image.

But for the images where we have many objects, we use the following method — For each class of objects, like (chair, person, car,….), we compute the average precision. The average precision score is the area under the Precision-Recall curve. Precision and recall values are between 0 and 1. So the AP falls between zero and one.

Average Precision

Precision-Recall curve

mAP Score(Mean Average Precision)- For each class, we have average precision values. So we take the mean of all the average precision values.

4. The Trade-Off Between Speed and mAP-

In the tasks such as a self-driving car or real-time face Detection systems, the speed of prediction is much more important, whereas in the case of Medical Data mAP is much more important. So out of all the algorithms which were classically used, YOLOv3 turned out to be the best. The speed and mAP of the different algorithms are as follows:

5. Darknet 53

YOLOv3 uses Fully Convolutional Network(FCN). YOLOv3 uses Darknet-53 as a backbone.

Convolutional Block uses three operations ie. Conv2D →Batch Normalization → LeakyRelu. For downsizing the image this network uses strides. So if we change the stride value then the extracted feature will be of the size ((n-k+2p/s)+1)*((n-k+2p/s)+1). where n is the size of the image, k is the size of the filter, p is padding size and s is the stride.

uses three operations ie. Conv2D →Batch Normalization → LeakyRelu. For downsizing the image this network uses strides. So if we change the stride value then the extracted feature will be of the size ((n-k+2p/s)+1)*((n-k+2p/s)+1). where n is the size of the image, k is the size of the filter, p is padding size and s is the stride. Residual Block performs subsequent convolution operations but the size of the kernel is 1*1 for the first convolution and 3*3 for the next convolution.

performs subsequent convolution operations but the size of the kernel is 1*1 for the first convolution and 3*3 for the next convolution. This is very easy to understand. Consider layers in each rectangle as a residual block. The whole network is a chain of multiple blocks with some strides 2 Conv layers in between to reduce dimension. Inside the block, there’s just a bottleneck structure (1x1 followed by 3x3) plus a skip connection.

We consider the output of the Darknet-53 from the last residual block.

One thing to observe in this architecture is that there are no pooling layers involved in this whole architecture of feature extractor Darknet-53. That's why it is called Fully Convolutional Network(FCN).

Darknet-53 was pre-trained on the ImageNet dataset.

6. Bounding Box and Output

At the end of Darknet-53, we have the output which is of the size 13*13*1024. We consider 13*13 cells. Each cell is represented as

Co-ordinates

Cell Representation

Here P0 is the objectness score which says if there is an object or not and P(Ci) = P0 * Pi

We pick maximum P(Ci) and assign that class to that object but there could be multiple boxes representing the same object So we multiply this whole above representation by B, where B is the Total Number of Boxes.

Prediction of tx, ty, tw, th was complex, so the concept of anchor boxes was introduced.

was introduced. Anchor Boxes are predefined boxes which are represented as :[Cx, Cy, Pw, Ph]

Nomenclature is:

(bx, by, bw, bh) → Final Bounding Box Co-ordinates (Cx, Cy, Pw, Ph) → Anchor Box Co-ordinates (Predefined) (tx, ty, tw, th) → It's what we represent in the BB coordinates and which are learned while training.

Anchor Box Selection on Custom Dataset From Research Paper

Final Bounding Box Co-ordinates

The model learns tx, ty, tw, th.

We have predefined Cx, Cy, Pw, Ph.

Using these two we can calculate bx, by, by, bh using the formula shown above.

YOLO v3, in total, uses 9 anchor boxes. Three for each scale. If you’re training YOLO on your own dataset, you should go about using K-Means clustering to generate 9 anchors. Then, arrange the anchors in descending order of a dimension. Assign the three biggest anchors for the first scale, the next three for the second scale, and the last three for the third.

7. Per-Class Sigmoid-

P0 → Object is present or not in an image.

P1, P2,…, Pc → Probability of each class of objects.

YOLO v3 now performs multilabel classification for objects detected in images. Earlier in YOLOv1 and v2, authors used to softmax the class scores and take the class with the maximum score to be the class of the object contained in the bounding box. This has been modified in YOLO v3. Softmaxing classes rest on the assumption that classes are mutually exclusive, or in simple words, if an object belongs to one class, then it cannot belong to the other. This works fine in the COCO dataset.

However, when we have classes like Person and Women in a dataset, then the above assumption fails. This is the reason why the authors of YOLOv3 have refrained from using softmax to get the classes. Instead, each class score is predicted using logistic regression and a threshold is used to predict multiple labels for an object. Classes with scores higher than this threshold are assigned to the box.

8. Loss Function-

The most important thing to know about any deep learning algorithm is the Loss Function. The loss function for YOLOv3 is complex so I am writing that down on the paper

Loss Function of YOLOv3

For images containing objects:

Loss contains the term which penalizes objectness score P0. It also involves the log-loss for each predicted class P(Ci). And it includes squared loss on the learned parameters like (sqrt(tw), sqrt(th), tx, and ty)* lambda_co-ord.

For images that do not contain objects:

log-loss for P0 * lamda_no_obj.

Typical values of labda_co-ord is 5 and lambda_no_obj is 0.5.

9. Multi-Scale Prediction

Instead of just using the 13*13 grid cells we can also use the 26*26 and 52*52 cells. So this idea of multi-scale prediction is taken from the Feature Pyramid Network (FPN).

Feature Vector For multi-scale prediction

The 13 x 13 layer is responsible for detecting large objects, whereas the 52 x 52 layer detects the smaller objects, with the 26 x 26 layer detecting medium objects. Here is a comparative analysis of different objects picked in the same object by different layers.

Outputs from different scales

10. Combining Boxes From Various Scale-

Combining the Different Scale

Filtering: Boxes with max(P0, P(Ci)) <0.5 are ignored as a filtering process. So they are not considered as an output prediction. Non-Max Suppression — Pick the box with the max IOU scores out of multiple boxes for one object.

11. Total Working of YOLOv3-

Complete Architecture of YOLOv3

Image is passed through the Darknet-53 network(Shown in Grey region in the above figure) which contains some sets of convolutional and residual blocks as discussed in the earlier section. Stride 2 convolution is used in between those blocks to reduce the size of the feature map and we take out 3 feature vectors from that network each of sizes 13x13, 26x26, 52x52.

The last output of Darknet-53 is convolved with the filter of size (255,1,1) to get the scale 3 feature map. Now scale 3 feature map output which is shown in the Blue region in the above figure is of the size (batch_size x13x13x255) and used to detect the big objects.

For scale 2 feature map(Pink Region in the figure), It gets two inputs one from the darknet-53 second last residual layer, and one from the upsampled output of the last stage of Darknet-53 which is concatenated and passed through two more convolution filters to get the output of scale 2. Scale 2 feature map’s output size is (26x26x255) and is useful in detecting medium-sized objects.

For scale-1 feature map(Orange Region in the figure), It gets two inputs one from the scale-2 stage and the third last residual block from the Darknet-53. These two inputs are concatenated and then passed through some convolution filters to get the output feature map of size (batch_size x 52x52x255). It is used to detect small objects.

For an image of size 416 x 416, YOLO predicts ((52 x 52) + (26 x 26) + 13 x 13)) x 3 = 10647 bounding boxes. However, in the case of our image, there’s only one object. So, you may ask how do we reduce the detections from 10647 to 1?

First, we filter boxes based on their objectness score. Generally, boxes having scores below a threshold (for example below 0.5) are ignored. Next, Non-maximum Suppression (NMS) intends to cure the problem of multiple detections of the same image. For example, all the 3 bounding boxes of the red grid cell may detect a box or the adjacent cells may detect the same object, so NMS is used to remove multiple detections.

Finally, we get the bounding box for an object in an image

12. Conclusion-

That’s it for the theory part. In the next blog, I will be writing about the Deep Sort algorithm for Object Tracking, and in the next blog following that, I will write about how to implement both these Object Detection and Object Tracking. I will soon put my code on my GitHub after writing my implementation blog. If you have any doubts About this, please comment or feel free to contact me via LinkedIn.

Thank you for Reading!!!!

13. References —