Build your Realtime custom object detector using YOLOv3
By: Surajit Saikia
Tags: Machine Learning, Yolov3, Realtime, Object Detection, Deep Learning

In this post, we will learn to build a real-time object detector using YOLOv3 network step by step. If you already know the details about the architecture, and you are more curious about the code, then you can directly start implementing, or else you can go through this paper to give a read.

Here, we will be using a subset of images from the Google open Image dataset (OID) to train the YOLOv3. OID is the largest existing dataset with object location annotations with 16 million bounding boxes for 600 object categories. Due to this, we will do our training using the best available large scale object detection dataset. If you want to know about this dataset, then here you can find more details. Once we train the YOLOv3 network, we will use our webcam for real-time object detection.

We will be downloading images corresponding to the following five classes: ‘Person’, ‘Car’, ‘Human face’, ‘Laptop’ and ’Mobile phone’, and then we will detect those objects in realtime. Let's start now.

Step:1 Install the dependencies and download the Repository

Step:2 Download the Dataset

Step:3 Traning using Custom images

Step: 4 Real-time Object detection using camera feed.

1. Install the dependencies and download the Repository

Please download the Repository from the following GitHub link, and install the dependencies by typing the following command from the root directory.

pip install -r requirements.txt

*Note: You need python version > =3.6 and a GPU machine with the recent version of CUDA and Tensorflow Installed. I have tested using TensorFlow 2.3 and Cuda version 11.1.

2. Downloading a subset of Google Open Dataset

To train the model, at first, we need to download a few classes of the dataset, and then we will annotate them in a proper format. For convenience, you can open a Jupyter notebook, and you can follow the code below to download and annotate the images.

1. Import the libraries

import cv2

from matplotlib import pyplot as plt

import numpy as np

import os

import pandas as pd

import random

from skimage import io

from shutil import copyfile

import sys

import time

2. Download the following three CSV files from this link.

train-images-boxable.csv This file contains the image name along with its URL

This file contains the image name along with its URL oidv6-train-annotations-bbox.csv This file contains the bounding box information with the image id and its label

This file contains the bounding box information with the image id and its label class-descriptions-boxable.csv This file contains the image label name corresponding to its class name

After downloading them assign the paths to variables.

imageboxes_file_name = './image_ids_and_rotation.csv'

annotationboxes_file_name = './oidv6-train-annotations-bbox.csv'

class_file_name = './class-descriptions-boxable.csv'

3. Read the CSV files using Pandas. (If you don’t have pandas then you can simply install by typing:

pip install pandas in the terminal

!pip install pandas in the Jupyter shell.

# READ THE CSV files

images_boxes = pd.read_csv(imagesboxes_file_name)

annotations_boxes = pd.read_csv(annotationboxes_file_name)

classes=pd.read_csv(class_file_name, header=None)

4. Check if the CSV files are properly loaded or not. If loaded, it will display the first five rows with respect to each of the CSV file

images_boxes.head()

annotations_boxes.head()

classes.head()

5. Next, define the classes that you want to download for training. Here, I am going to select ‘Person’, ‘Car’, ‘Human face’, ‘Laptop’ and ’Mobile phone’. But, you are free to choose any of the 600 classes that are present in the Google Open Image Dataset.

classes.loc[classes[1].isin(['Person', 'Car', 'Human face', 'Laptop','Mobile phone' ])]

You will see the classes that you have selected

6. Create Labels

person_pd = class_descriptions[class_descriptions[1]=='Person']

face_pd = class_descriptions[class_descriptions[1]=='Human face']

car_pd = class_descriptions[class_descriptions[1]=='Car']

mobile_pd = class_descriptions[class_descriptions[1]=='Mobile phone']

laptop_pd = class_descriptions[class_descriptions[1]=='Laptop'] label_name_person = person_pd[0].values[0]

label_name_face = face_pd[0].values[0]

label_name_car = car_pd[0].values[0]

label_name_mobile = mobile_pd[0].values[0]

label_name_laptop = laptop_pd[0].values[0]

7. Create bounding box list for each of the image labels.

#Bounding of annotation of each of the mentioned classess

person_bbox = annotations_bbox[annotations_bbox['LabelName']==label_name_person]

face_bbox = annotations_bbox[annotations_bbox['LabelName']==label_name_face]

car_bbox = annotations_bbox[annotations_bbox['LabelName']==label_name_car]

laptop_bbox = annotations_bbox[annotations_bbox['LabelName']==label_name_laptop]

mobile_bbox = annotations_bbox[annotations_bbox['LabelName']==label_name_mobile] person_id = person_bbox['ImageID']

face_id = face_bbox['ImageID']

car_id = car_bbox['ImageID']

laptop_id = laptop_bbox['ImageID']

mobile_id = mobile_bbox['ImageID']

8. Set the Number of images that you want to download with respect to each of the classes. Here, I set n=1500, and you can change it. However, it depends on the number of images that are present originally in the dataset.

# set n=1500, to select 1500 images for each of the classes

n = 1500

subperson_id = random.sample(list(person_id), n)

subface_id = random.sample(list(face_id), n)

subcar_id = random.sample(list(car_id), n)

submobile_id = random.sample(list(mobile_id), n)

sublaptop_id = random.sample(list(laptop_id), n) subperson_pd = images_boxable.loc[images_boxable['ImageID'].isin(subperson_id)]

subface_pd = images_boxable.loc[images_boxable['ImageID'].isin(subface_id)]

subcar_pd = images_boxable.loc[images_boxable['ImageID'].isin(subcar_id)]

submobile_pd = images_boxable.loc[images_boxable['ImageID'].isin(submobile_id)]

sublaptop_pd = images_boxable.loc[images_boxable['ImageID'].isin(sublaptop_id)]

9. Set up the URL link to download the selected images, and define the classes in the same order as the list containing the object dictionaries.

subperson_dict = subperson_pd[["ImageID", "OriginalURL"]].set_index('ImageID')["OriginalURL"].to_dict()

subface_dict = subface_pd[["ImageID", "OriginalURL"]].set_index('ImageID')["OriginalURL"].to_dict()

subcar_dict = subcar_pd[["ImageID", "OriginalURL"]].set_index('ImageID')["OriginalURL"].to_dict()

sublaptop_dict = sublaptop_pd[["ImageID", "OriginalURL"]].set_index('ImageID')["OriginalURL"].to_dict()

submobile_dict = submobile_pd[["ImageID", "OriginalURL"]].set_index('ImageID')["OriginalURL"].to_dict() lis = [subperson_dict, subface_dict, subcar_dict, submobile_dict, sublaptop_dict]

classes = ['Person', 'Human face', 'Car', 'Mobile phone','Laptop']

10. Download the Dataset: The following code will create folders with respect to each of the classes that you have selected, and the images will get saved accordingly.

for idx, cls in enumerate(classes):

dire = './Data/Train/'+cls

if not os.path.exists(dire):

os.mkdir(dire)

for img_id, url in lis[idx].items():

try:

img = io.imread(url)

saved_path = os.path.join(dire, img_id+".jpg")

io.imsave(saved_path, img)

except Exception as e:

print('handled')

11. Split the downloaded images into train and test images, and save them in the Train and Test folder.

# save images to train and test directory train_path = './Data/Train/'

test_path = './Data/Test/' for i in range(len(classes)):

all_imgs = os.listdir(classes[i])

all_imgs = [f for f in all_imgs if not f.startswith('.')]

random.shuffle(all_imgs)



# train: 0.75 validation: 0.25

split= int(n*0.75) train_imgs = all_imgs[:split] # 75 percent images

test_imgs = all_imgs[split:] # 25 percent images



# copy to train directory

for j in range(len(train_imgs)):

original_path = os.path.join(classes[i], train_imgs[j])

new_path = os.path.join(train_path, train_imgs[j])

copyfile(original_path, new_path)



# copy to test directory

for j in range(len(test_imgs)):

original_path = os.path.join(classes[i], test_imgs[j])

new_path = os.path.join(test_path, test_imgs[j])

copyfile(original_path, new_path)

12. Create Training data: The following code will parse the training images and the bounding boxes. Similarly, you can do for the test images.

# create training data: Check the labels below

label_names = [label_name_person, label_name_face,label_name_car, label_name_mobile, label_name_laptop] train_df = pd.DataFrame(columns=['FileName', 'XMin', 'YMin', 'XMax', 'YMax', 'ClassName']) # Find boxes in each image and put them in a dataframe

train_imgs = os.listdir(train_path)

train_imgs = [name for name in train_imgs if not name.startswith('.')] for i in range(len(train_imgs)):

sys.stdout.write('Parse train_imgs ' + str(i) + '; Number of boxes: ' + str(len(train_df)) + '\r')

sys.stdout.flush()

img_name = train_imgs[i]

img_id = img_name[0:16]

tmp_df = annotations_bbox[annotations_bbox['ImageID']==img_id]

for index, row in tmp_df.iterrows():

labelName = row['LabelName']

for i in range(len(label_names)):

if labelName == label_names[i]:

train_df = train_df.append({'FileName': img_name,

'XMin': row['XMin'],

'YMin': row['YMin'],

'XMax': row['XMax'],

'YMax': row['YMax'],

'ClassName': classes[i]},

ignore_index=True)

You can see the structure of the training data by typing:

train_df.head()



13. Now, since everything is correct, we convert the training data into CSV format and then we create the “trainannotation.txt” file. Also, you need to define the class names in this file. For example, if you want to train your model to detect ‘Person’, ‘Human face’, ‘Car’, ‘Mobile phone’ and ‘Laptop’, then introduce them in the data_classes.txt file.

# Create Dictionary of class names

classes_dict = {'Person':0, 'Human face':1, 'Car':2, 'Mobile phone':3,'Laptop':4} # mention the file path where the annotation file needs to look for # the images

# create the annotation file for training

train_df = pd.read_csv('train.csv')



# for training

with open("trainannotation.txt", "w+") as f:

for idx, row in train_df.iterrows():

img = cv2.imread(train_path + row['FileName'])

height, width = img.shape[:2]

x1 = int(row['XMin'] * width)

y1 = int(row['YMin'] * height)

x2 = int(row['XMax'] * width)

y2 = int(row['YMax'] * height)



file_path = 'train'

fileName = os.path.join(train_path, row['FileName'])

className = row['ClassName']

print(fileName + ' ' + str(x1) + ',' + str(y1) + ',' + str(x2) + ',' + str(y2) + ',' + str(classes_dict[className]))

f.write(fileName + ' ' + str(x1) + ',' + str(y1) + ',' + str(x2) + ',' + str(y2) + ',' + str(classes_dict[className]) + '

')

Now you have the training data available with properly annotated. Also, you can use this code to generate annotated data for training other object detection frameworks such as YOLOv2, FasterRCNN and YOLOv4.

3. Training the Module using the downloaded images

Now, your training images and annotations files are saved in their respective folders. You can now train the model just by typing the following from the root directory.

python ./Training/TrainerOID.py

The TrainOID.py file consists of various command-line options which you can modify in order to change epochs, learning rate, batch size etc. To speed up training, it is recommended to use a GPU with CUDA support.

4. Deploying for Real-time Object detection using Webcam.

Once you have trained the model, you will be needing a webcam to see how well the objects are detected. The trained model will be saved as trained_weights_ODI.h5 in Model_weights folder.

To test the model, keep the webcam ready and just type the following command from the root directory.

python ./Deployment/videoRT.py

Now, you should be seeing the selected objects being detected in real-time.

Screenshot during real-time object detection using a web camera.

References: