Object Detection with YOLOv3 using Keras | by Renu Khandelwal | Towards Data Science

Part 1- CNN, R-CNN, Fast R-CNN, Faster R-CNN
Part 2 —Understanding YOLO, YOLOv2, YOLO v3
This is the third article in the series where we will predict the bounding boxes and classes using YOLOv3. Code available at github
You only look once (YOLO) at an image to predict what objects are present and where they are present using a single convolutional network. YOLO predicts multiple bounding boxes and class probabilities for those boxes.
This code will use pre-trained weights from yolo v3 and then predict the bounding boxes and class probabilities using keras library
The code is strongly inspired by experiencor’s keras-yolo3 project for performing object detection with a YOLOv3 model. Code is broken code into simple steps to predict the bounding boxes and classes using yolov3 model. The original code is available at github from Huynh Ngoc Anh.Yolo3 pre-trained weights can be downloaded from YOLOv3 pre-trained weights.
YOLOv3 model uses pre-trained weights for standard object detection problems such as a kangaroo dataset, racoon dataset, red blood cell detection, and others. This model will be used for object detection on new images.
Step 1: Importing the required libraries
import osimport scipy.ioimport scipy.miscimport numpy as npimport pandas as pdimport PILimport structimport cv2from numpy import expand_dimsimport tensorflow as tffrom skimage.transform import resizefrom keras import backend as Kfrom keras.layers import Input, Lambda, Conv2D, BatchNormalization, LeakyReLU, ZeroPadding2D, UpSampling2Dfrom keras.models import load_model, Modelfrom keras.layers.merge import add, concatenatefrom keras.preprocessing.image import load_imgfrom keras.preprocessing.image import img_to_arrayimport matplotlib.pyplot as pltfrom matplotlib.pyplot import imshowfrom matplotlib.patches import Rectangle%matplotlib inline
Step 2: Create a class WeightReader to load the pre-trained weights for yolov3
WeightReader class will parse the file and load the model weights into memory to set it in our Keras model.
# class to load the pretrained Weightsclass WeightReader:    def __init__(self, weight_file):        with open(weight_file, 'rb') as w_f:            major,    = struct.unpack('i', w_f.read(4))            minor,    = struct.unpack('i', w_f.read(4))            revision, = struct.unpack('i', w_f.read(4))if (major*10 + minor) >= 2 and major < 1000 and minor < 1000:                w_f.read(8)            else:                w_f.read(4)transpose = (major > 1000) or (minor > 1000)                        binary = w_f.read()self.offset = 0        self.all_weights = np.frombuffer(binary, dtype='float32')            def read_bytes(self, size):        self.offset = self.offset + size        return self.all_weights[self.offset-size:self.offset]def load_weights(self, model):        for i in range(106):            try:                conv_layer = model.get_layer('conv_' + str(i))                print("loading weights of convolution #" + str(i))if i not in [81, 93, 105]:                    norm_layer = model.get_layer('bnorm_' + str(i))size = np.prod(norm_layer.get_weights()[0].shape)beta  = self.read_bytes(size) # bias                    gamma = self.read_bytes(size) # scale                    mean  = self.read_bytes(size) # mean                    var   = self.read_bytes(size) # varianceweights = norm_layer.set_weights([gamma, beta, mean, var])if len(conv_layer.get_weights()) > 1:                    bias   = self.read_bytes(np.prod(conv_layer.get_weights()[1].shape))                    kernel = self.read_bytes(np.prod(conv_layer.get_weights()[0].shape))                                        kernel = kernel.reshape(list(reversed(conv_layer.get_weights()[0].shape)))                    kernel = kernel.transpose([2,3,1,0])                    conv_layer.set_weights([kernel, bias])                else:                    kernel = self.read_bytes(np.prod(conv_layer.get_weights()[0].shape))                    kernel = kernel.reshape(list(reversed(conv_layer.get_weights()[0].shape)))                    kernel = kernel.transpose([2,3,1,0])                    conv_layer.set_weights([kernel])            except ValueError:                print("no convolution #" + str(i))             def reset(self):        self.offset = 0
Step 3: Create the Yolo v3 model.
We first create a function for creating the Convolutional blocks
def _conv_block(inp, convs, skip=True):    x = inp    count = 0        for conv in convs:        if count == (len(convs) - 2) and skip:            skip_connection = x        count += 1                if conv['stride'] > 1: x = ZeroPadding2D(((1,0),(1,0)))(x) # peculiar padding as darknet prefer left and top        x = Conv2D(conv['filter'],                    conv['kernel'],                    strides=conv['stride'],                    padding='valid' if conv['stride'] > 1 else 'same', # peculiar padding as darknet prefer left and top                   name='conv_' + str(conv['layer_idx']),                    use_bias=False if conv['bnorm'] else True)(x)        if conv['bnorm']: x = BatchNormalization(epsilon=0.001, name='bnorm_' + str(conv['layer_idx']))(x)        if conv['leaky']: x = LeakyReLU(alpha=0.1, name='leaky_' + str(conv['layer_idx']))(x)return add([skip_connection, x]) if skip else x
Next, we create a Darknet with 108 convolutional layers. I am running it on CPU, On a GPU it is almost 500 times faster
# creating the YOLO modeldef make_yolov3_model():    input_image = Input(shape=(None, None, 3))# Layer  0 => 4    x = _conv_block(input_image, [{'filter': 32, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 0},                                  {'filter': 64, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 1},                                  {'filter': 32, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 2},                                  {'filter': 64, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 3}])# Layer  5 => 8    x = _conv_block(x, [{'filter': 128, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 5},                        {'filter':  64, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 6},                        {'filter': 128, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 7}])# Layer  9 => 11    x = _conv_block(x, [{'filter':  64, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 9},                        {'filter': 128, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 10}])# Layer 12 => 15    x = _conv_block(x, [{'filter': 256, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 12},                        {'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 13},                        {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 14}])# Layer 16 => 36    for i in range(7):        x = _conv_block(x, [{'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 16+i*3},                            {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 17+i*3}])            skip_36 = x            # Layer 37 => 40    x = _conv_block(x, [{'filter': 512, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 37},                        {'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 38},                        {'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 39}])# Layer 41 => 61    for i in range(7):        x = _conv_block(x, [{'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 41+i*3},                            {'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 42+i*3}])            skip_61 = x            # Layer 62 => 65    x = _conv_block(x, [{'filter': 1024, 'kernel': 3, 'stride': 2, 'bnorm': True, 'leaky': True, 'layer_idx': 62},                        {'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 63},                        {'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 64}])# Layer 66 => 74    for i in range(3):        x = _conv_block(x, [{'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 66+i*3},                            {'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 67+i*3}])            # Layer 75 => 79    x = _conv_block(x, [{'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 75},                        {'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 76},                        {'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 77},                        {'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 78},                        {'filter':  512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 79}], skip=False)# Layer 80 => 82    yolo_82 = _conv_block(x, [{'filter': 1024, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 80},                              {'filter':  255, 'kernel': 1, 'stride': 1, 'bnorm': False, 'leaky': False, 'layer_idx': 81}], skip=False)# Layer 83 => 86    x = _conv_block(x, [{'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 84}], skip=False)    x = UpSampling2D(2)(x)    x = concatenate([x, skip_61])# Layer 87 => 91    x = _conv_block(x, [{'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 87},                        {'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 88},                        {'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 89},                        {'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 90},                        {'filter': 256, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 91}], skip=False)# Layer 92 => 94    yolo_94 = _conv_block(x, [{'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 92},                              {'filter': 255, 'kernel': 1, 'stride': 1, 'bnorm': False, 'leaky': False, 'layer_idx': 93}], skip=False)# Layer 95 => 98    x = _conv_block(x, [{'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True,   'layer_idx': 96}], skip=False)    x = UpSampling2D(2)(x)    x = concatenate([x, skip_36])# Layer 99 => 106    yolo_106 = _conv_block(x, [{'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 99},                               {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 100},                               {'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 101},                               {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 102},                               {'filter': 128, 'kernel': 1, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 103},                               {'filter': 256, 'kernel': 3, 'stride': 1, 'bnorm': True,  'leaky': True,  'layer_idx': 104},                               {'filter': 255, 'kernel': 1, 'stride': 1, 'bnorm': False, 'leaky': False, 'layer_idx': 105}], skip=False)model = Model(input_image, [yolo_82, yolo_94, yolo_106])        return model
Step 4: we now create the yolo model and load the pre-trained weights
# create the yolo v3yolov3 = make_yolov3_model()# load the weights trained on COCO into the modelweight_reader = WeightReader(‘yolov3.weights’)weight_reader.load_weights(yolov3)
Step 5: Setting up the variables.
Input image size for Yolov3 is 416 x 416 which we set using net_h and net_w.
Object threshold is set to 0.5 and Non-max suppression threshold is set to 0.45
We set the anchor boxes and then define the 80 labels for the Common Objects in Context (COCO) model to predict
net_h, net_w = 416, 416obj_thresh, nms_thresh = 0.5, 0.45anchors = [[116,90, 156,198, 373,326], [30,61, 62,45, 59,119], [10,13, 16,30, 33,23]]labels = [“person”, “bicycle”, “car”, “motorbike”, “aeroplane”, “bus”, “train”, “truck”, “boat”, “traffic light”, “fire hydrant”, “stop sign”, “parking meter”, “bench”, “bird”, “cat”, “dog”, “horse”, “sheep”, “cow”, “elephant”, “bear”, “zebra”, “giraffe”, \ “backpack”, “umbrella”, “handbag”, “tie”, “suitcase”, “frisbee”, “skis”, “snowboard”, “sports ball”, “kite”, “baseball bat”, “baseball glove”, “skateboard”, “surfboard”, “tennis racket”, “bottle”, “wine glass”, “cup”, “fork”, “knife”, “spoon”, “bowl”, “banana”, “apple”, “sandwich”, “orange”, “broccoli”, “carrot”, “hot dog”, “pizza”, “donut”, “cake”, “chair”, “sofa”, “pottedplant”, “bed”, “diningtable”, “toilet”, “tvmonitor”, “laptop”, “mouse”, \ “remote”, “keyboard”, “cell phone”, “microwave”, “oven”, “toaster”, “sink”, “refrigerator”, “book”, “clock”, “vase”, “scissors”, “teddy bear”, “hair drier”, “toothbrush”]
Step 6: Loading the image into the right input shape of 416 x 416
from numpy import expand_dimsdef load_image_pixels(filename, shape): # load the image to get its shape image = load_img(filename) width, height = image.size # load the image with the required size image = load_img(filename, target_size=shape) # convert to numpy array image = img_to_array(image) # scale pixel values to [0, 1] image = image.astype(‘float32’) image /= 255.0 # add a dimension so that we have one sample image = expand_dims(image, 0) return image, width, height
Step 7: Create a class for the Bounding Box.
BoundBox defines the corners of each bounding box in the context of the input image shape and class probabilities.
class BoundBox: def __init__(self, xmin, ymin, xmax, ymax, objness = None, classes = None): self.xmin = xmin self.ymin = ymin self.xmax = xmax self.ymax = ymax  self.objness = objness self.classes = classesself.label = -1 self.score = -1def get_label(self): if self.label == -1: self.label = np.argmax(self.classes)  return self.label  def get_score(self): if self.score == -1: self.score = self.classes[self.get_label()]  return self.score
Step 8: Define functions for
Interval overlap — checks if two intervals overlap. Two intervals do not overlap when one ends before the other begins.
Intersection over Union(IoU) of two boxes
Non-Max Suppression which takes boxes that have the presence of objects in them along with non-max threshold as a parameter
Sigmoid function
def _sigmoid(x): return 1. / (1. + np.exp(-x))def _interval_overlap(interval_a, interval_b): x1, x2 = interval_a x3, x4 = interval_bif x3 < x1: if x4 < x1: return 0 else: return min(x2,x4) — x1 else: if x2 < x3: return 0 else: return min(x2,x4) — x3 def bbox_iou(box1, box2): intersect_w = _interval_overlap([box1.xmin, box1.xmax], [box2.xmin, box2.xmax]) intersect_h = _interval_overlap([box1.ymin, box1.ymax], [box2.ymin, box2.ymax])  intersect = intersect_w * intersect_hw1, h1 = box1.xmax-box1.xmin, box1.ymax-box1.ymin w2, h2 = box2.xmax-box2.xmin, box2.ymax-box2.ymin  union = w1*h1 + w2*h2 — intersect  return float(intersect) / uniondef do_nms(boxes, nms_thresh): if len(boxes) > 0: nb_class = len(boxes[0].classes) else: return  for c in range(nb_class): sorted_indices = np.argsort([-box.classes[c] for box in boxes])for i in range(len(sorted_indices)): index_i = sorted_indices[i]if boxes[index_i].classes[c] == 0: continuefor j in range(i+1, len(sorted_indices)): index_j = sorted_indices[j]if bbox_iou(boxes[index_i], boxes[index_j]) >= nms_thresh: boxes[index_j].classes[c] = 0
Step 9: Decode the output of the network.
We will iterate through each of the each one of the NumPy arrays, one at a time and decode the candidate bounding boxes and class predictions based on the object threshold.
The first 4 elements will be the coordinates of the Bounding box, 5th element will be object score followed by the class probabilities
def decode_netout(netout, anchors, obj_thresh, net_h, net_w): grid_h, grid_w = netout.shape[:2] nb_box = 3 netout = netout.reshape((grid_h, grid_w, nb_box, -1)) nb_class = netout.shape[-1] — 5boxes = [] netout[..., :2] = _sigmoid(netout[..., :2]) netout[..., 4:] = _sigmoid(netout[..., 4:]) netout[..., 5:] = netout[..., 4][..., np.newaxis] * netout[..., 5:] netout[..., 5:] *= netout[..., 5:] > obj_threshfor i in range(grid_h*grid_w): row = i / grid_w col = i % grid_w  for b in range(nb_box): # 4th element is objectness score objectness = netout[int(row)][int(col)][b][4] #objectness = netout[..., :4]  if(objectness.all() <= obj_thresh): continue  # first 4 elements are x, y, w, and h x, y, w, h = netout[int(row)][int(col)][b][:4]x = (col + x) / grid_w # center position, unit: image width y = (row + y) / grid_h # center position, unit: image height w = anchors[2 * b + 0] * np.exp(w) / net_w # unit: image width h = anchors[2 * b + 1] * np.exp(h) / net_h # unit: image height   # last elements are class probabilities classes = netout[int(row)][col][b][5:]  box = BoundBox(x-w/2, y-h/2, x+w/2, y+h/2, objectness, classes) boxes.append(box)return boxes
Step 10: Correcting the Yolo boxes.
We have the bounding boxes but they need to be stretched back into the shape of the original image. This will allow plotting the original image and draw the bounding boxes, detecting real objects.
def correct_yolo_boxes(boxes, image_h, image_w, net_h, net_w): if (float(net_w)/image_w) < (float(net_h)/image_h): new_w = net_w new_h = (image_h*net_w)/image_w else: new_h = net_w new_w = (image_w*net_h)/image_h  for i in range(len(boxes)): x_offset, x_scale = (net_w — new_w)/2./net_w, float(new_w)/net_w y_offset, y_scale = (net_h — new_h)/2./net_h, float(new_h)/net_h  boxes[i].xmin = int((boxes[i].xmin — x_offset) / x_scale * image_w) boxes[i].xmax = int((boxes[i].xmax — x_offset) / x_scale * image_w) boxes[i].ymin = int((boxes[i].ymin — y_offset) / y_scale * image_h) boxes[i].ymax = int((boxes[i].ymax — y_offset) / y_scale * image_h)
Step 11: Get all boxes above the specified threshold.
get_boxes function takes the list of boxes, labels, and threshold as arguments and returns parallel lists of boxes, labels, and scores.
def get_boxes(boxes, labels, thresh):    v_boxes, v_labels, v_scores = list(), list(), list()    # enumerate all boxes    for box in boxes:        # enumerate all possible labels        for i in range(len(labels)):            # check if the threshold for this label is high enough            if box.classes[i] > thresh:                v_boxes.append(box)                v_labels.append(labels[i])                v_scores.append(box.classes[i]*100)                # don't break, many labels may trigger for one box    return v_boxes, v_labels, v_scores
Step 12: Drawing a white box around the object present in the image.
from matplotlib.patches import Rectangledef draw_boxes(filename, v_boxes, v_labels, v_scores): # load the image data = plt.imread(filename) # plot the image plt.imshow(data) # get the context for drawing boxes ax = plt.gca() # plot each box for i in range(len(v_boxes)):box = v_boxes[i]        # get coordinates        y1, x1, y2, x2 = box.ymin, box.xmin, box.ymax, box.xmax        # calculate width and height of the box        width, height = x2 - x1, y2 - y1        # create the shape        rect = Rectangle((x1, y1), width, height, fill=False, color='red')        # draw the box        ax.add_patch(rect)        # draw text and score in top left corner        label = "%s (%.3f)" % (v_labels[i], v_scores[i])        plt.text(x1, y1, label, color='red')    # show the plotplt.show()
Finally, we tie the code to make the prediction on a new image.
First we load the image into the input shape of 416 x 416 using load_image_pixels() function.
Predict the boxes suing yolov3 predict () method
Yolov3 model will be predicting multiple boxes for the same object.
We then decode the candidate bounding boxes and class predictions based on the object threshold using decode_netout() function
we correct the bounding boxes by using correct_yolo_boxes() function to be stretched back into the shape of the original image
Bounding boxes will be filtered based on the overlap defined by IoU and then applying the non-max suppression using the do_nms() function.
we get the boxes using get_boxes() which are above the specified class threshold of 0.6 and the then plot the boxes using draw_boxes() function on the image.
# define our new imagephoto_filename = 'eagle.png'# load and prepare imageimage, image_w, image_h = load_image_pixels(photo_filename, (net_w, net_w))# make predictionyolos = yolov3.predict(image)# summarize the shape of the list of arraysprint([a.shape for a in yolos])# define the anchorsanchors = [[116,90, 156,198, 373,326], [30,61, 62,45, 59,119], [10,13, 16,30, 33,23]]# define the probability threshold for detected objectsclass_threshold = 0.6boxes = list()for i in range(len(yolos)):        # decode the output of the network    boxes += decode_netout(yolos[i][0], anchors[i], obj_thresh, nms_thresh, net_h, net_w)# correct the sizes of the bounding boxescorrect_yolo_boxes(boxes, image_h, image_w, net_h, net_w)# suppress non-maximal boxesdo_nms(boxes, nms_thresh)# get the details of the detected objectsv_boxes, v_labels, v_scores = get_boxes(boxes, labels, class_threshold)# summarize what we foundfor i in range(len(v_boxes)):    print(v_labels[i], v_scores[i])# draw what we founddraw_boxes(photo_filename, v_boxes, v_labels, v_scores)
Original image
Image after plotting the bounding box and class
Code is available at Github
YOLO V3 paper by Joseph Redmon, Ali Farhadi