Teaching Cars To See — Vehicle Detection Using Machine Learning And Computer Vision | by Eddie Forson | Towards Data Science

This is the final project of Term 1 of the Udacity Self-Driving Car Engineer Nanodegree. You can find all code related to this project on github. You can also read my posts on previous projects:
project 1: Detecting Lane Lines Using Computer Vision
project 2: Traffic Sign Classification Using Deep Learning
project 3: Steering Angle Prediction Using Deep Learning
project 4: Advanced Lane Detection Using Computer Vision
When we drive, we constantly pay attention to our environment, as our safety and that of many other people are at stake. We particularly look out for position of potential obstacles, whether they be other cars, pedestrians, or objects on the road. Similarly, as we develop the intelligence and sensors necessary to power a autonomous vehicles, it is of the utmost importance that such vehicles can detect obstacles as well, as it reinforces the car’s understanding of its environment. One of the most important types of ostacles to detect is other vehicles on the road, as they would most likely be the biggest objects in our lane or neighbouring ones and therefore constitute a potential hazard.
A number of techniques for obstacle detection have been developed throughout the literature, from traditional computer vision techniques to deep learning ones, and more. In this exercise, we build a vehicle detector by employing a conventional computer vision technique called Histogram of Oriented Gradients (HOG), combined with a machine learning algorithm called Support Vector Machines (SVM).
Udacity generously provided a balanced dataset with the following characteristics:
~ 9K images of vehicles
~ 9K images of non-vehicles
all images are 64x64
The dataset comes from the GTI Vehicle Image Database, KITTI Vision Benchmark Suite, and examples extracted from the project video itself. The latter is much larger and was not used for this project. However, it would be a great addition in the future, especially as we plan to build a classifier using deep learning. You can see a sample of images from the dataset below:
We can clearly see both vehicle and non-vehicle images. Non-vehicle images tend to be other elements of the road such as the asphalt, road signs or pavement. The distinction is very clear. Most images also display the vehicle in the center, but in different orientations, which is good. Moreover, there is a good variety of car types and colors, as well as lighting conditions.
Using HOG for detection was popularised by Navneet Dalal and Bill Triggs after showing impressive results in their paper named Histogram of Oriented Gradients For Human Detection. The algorithm is well explained by Satya Mallick on this post, for those who want to acquire a stronger fundamental grasp of HOG.
We firstly explored different configurations for the following values in the HOG algorithm, on a RGB image:
number of orientations (denoted by o)
pixels per cell (denoted by px/c)
The cells per block were originally fixed at 2 (denoted by c/bk). The images below show the results obtained on the sample vehicle image in RGB format:
From pure observation, it looks like a HOG configuration with:
11 orientations
14 pixels per cell
2 cells per blocks
produces the most distinctive gradients of a vehicle. We have not experimented with different cells per block so let us try now.
To the human eye, there is no significant difference that we notice visually. We would ideally like to reduce the feature space for faster computation. We will settle for now on 3 cells per block.
We must now explore the most suitable color space for our configuration, as it seems our HOG features across the 3 RGB channels are too similar, therefore it feels we are not generating features with enough variations.
We generate the following outputs across a multitude of color spaces:
For some color channels, it is difficult to interpret the result of HOG. Interestingly, it seems the first color channel in YUV, YCrCb, and LAB could be enough to capture the gradients we are looking for. In HSV and HLS it is respectively on the Value and Lightness channels that HOG captures the most significant features for the vehicle.
To confirm our hypothesis, let us try with a different image of a vehicle:
Houston, we’ve have a problem here... On a dark image such as the one above, we can observe that HOG on the channel that carries the most light information produces bad results. We must therefore take into account all color channels to capture the most features. In the end, our configuration looks as follows:
ALL channels of YCrCb color space
HOG orientations of 11
HOG pixels per cell of 14
HOG cells per block of 2
We will also be adding color information to strengthen our feature set. To do so we simply produce histograms across all color channels, using 32 bins, as shown below:
def color_histogram(img, nbins=32, bins_range=(0, 256)):    """    Returns the histograms of the color image across all channels, as a concatenanted feature vector    """    # Compute the histogram of the color channels separately    channel1_hist = np.histogram(img[:,:,0], bins=nbins, range=bins_range)    channel2_hist = np.histogram(img[:,:,1], bins=nbins, range=bins_range)    channel3_hist = np.histogram(img[:,:,2], bins=nbins, range=bins_range)    # Concatenate the histograms into a single feature vector and return it    return np.concatenate((channel1_hist[0], channel2_hist[0], channel3_hist[0]))    
The classifier is responsible for categorising the images we submit into either vehicle or non-vehicle classes. To do so, we must take the following steps:
Load our images from the dataset
Extract the features we desire
Normalise those features
Split the dataset for training and testing
Build a classifier with the appropriate parameters
Train the classifier on training data
As discussed in the previous section, we have decided to only retain one feature: the HOG feature vector computed on the Y channel of our YCrCb image.
We randomly split our dataset, leaving 20% of it for testing. Moreover, we scale the data by employing a sklearn.preprocessing.StandardScaler normaliser.
We did not have enough time to experiment with many classifiers so opted to use Support Vector Machines (SVMs) as they are commonly combined with HOG in the literature for object detection problems. Moreover, we used a SVC with kernel rbf as it provided the best accuracy, while being slower than a LinearSVC. We accepted the tradeoff as the detection of the SVC with rbf kernel was much stronger when we tested it on a series of images.
The ideal parameters among kernel types (linear or rbf), C (1, 100, 1000, 1000) and gamma (auto, 0.01, 0.1, 1) were obtained by using the GridSearchCV function. The best configuration achieved over 99% accuracy and had the following parameters:
kernel = rbf
C = 100
gamma = auto
We created sliding windows of multiple dimensions, ranging from 64x64 to 256x256 pixels, to test portions of the image against the classifier and retained only positive predictions. We generally slid the bigger windows from the bottom of the screen, as this would correspond to where vehicles would appear the largest. Smaller windows would slide higher on the screen. Moreover, we have the ability to configure the cell overlap and have currently set it 1 for maximum coverage (i.e. overlap every 14 pixels * scale, where the minimum window of scale 1 is 64x64). We stop attempting to detect vehicles on anything below 350 pixels in the y direction (i.e. higher portion of the image on the screen). The image below shows the example of overlapping sliding windows with cell overlap set to 4:
The classifier sometimes misclassifies sections of the images that are actually not a vehicle. To avoid highlighting those on the video, we take advantage of the redundancy we created with our multi-size sliding windows and count the number of times our classifier predicted vehicle for a given section of the image across all the windows it appears in. We first label objects with overlapping windows using scipy.ndimage.measurements’ label function. We then extract the positions of each label by determining the smallest bounding box our detected object could fit in. We only retain sections of the image where the detected threshold is set to a particular value. From experimentation, we find out that a threshold of 4 is enough to attain solid results on the project video. The photo below illustrates how the heatmap and thresholding work:
The first mini heatmap represents the original raw detections from the classifier, whereas the second one shows the thresholded areas, where the intensity of the red increases as the number of overlapping windows goes up. The last mini image on the right shows all the windows where our classifier predicted vehicle. In this example we were in fact using a LinearSVC, which tends to get the prediction wrong more often than the rbf SVC.
To further strengthen our pipeline, we have decided to smoothen all detected windows every n frames. To do so, we accumulate all detected windows between frames (n-1)*f+1 to n*f, where n is a positive scalar that represents the group of frames we are in. We have created the following class that encapsulates a detected object:
class DetectedObject:    """    The DetectedObject class encapsulates information about an object identified by our detector    """    def __init__(self, bounding_box, img_patch, frame_nb):        self.bounding_box = bounding_box        self.img_patch = img_patch         self.frame_nb = frame_nb        self.centroid = (int((bounding_box[0][0] + bounding_box[1][0]) / 2), int((bounding_box[0][1] + bounding_box[1][1]) / 2))        self.similar_objects = []...
Everytime we detect a new object on the current or next frames in the group, we check whether we have detected a similar object in the past, and if so, we append the similar object, thus increasing this object’s count across multiple frames. At frame n*f we only retain detected objects (and their associated bounding boxes) that have over m detected counts, thereby achieving some kind of double filtering in the pipeline (the first filtering was the threshold on the number overlapping bounding boxes).
On the below gif, you can see that there is a split second between when we have a single bounding box covering both cars and when each car has its own bounding box: the frame aggregation logic must have waited until the two windows appeared enough times before displaying them:
The video link below shows a successful detection of vehicles. As this is the final project for Term 1, I just had to use the track End Titles from Tron Legacy’s OST as background music — nothing else would have been more fitting 😎. Enjoy, as usual!
This was a tricky project, especially for those who opted for the more conventional computer vision and machine learning approach as opposed to deep learning. The following steps were quite time consuming:
determining the most suitable features (HOG, image color histogram, etc)
exploring the combination of HOG parameters + color spaces
applying grid search to find the most suitable classifier
Moreover, in our pipeline we struggled with the following:
Determining correct position of our sliding windows and the overlap
Identifying suitable threshold for overlapping detection
Adopting suitable frame sampling rate
Finding a good enough minimum detection count over multiple frames
Aggregating the combined window dimensions for overlapping detections
The pipeline would fail for object that are not vehicles but detected as such by the classifier, and where such false detections occur over enough overlapping windows to break through the threshold configured, and do so consistently over minimum number of frames per group. The drawn bounding boxes do not always perfectly fit the vehicles and are being redrawn every n frames, therefore causing the impression of a lack of smoothness. Moreover, the frame aggregation could be improved by using a rolling window of n frames as opposed to batch aggregation every n frames.
The last problem is that our pipeline is too slow. We should not be sliding windows across the whole screen, only portions of the screen need to be looked at: in the future we could employ decision trees to identify regions of interest for instance. We could also look at reducing the number of sliding windows as well as employing a faster classifier like a LinearSVC to speed up detection (but accuracy decreases significantly too). Still, this vehicle detection pipeline is unlikely to work in real time.
In the future, a deep learning approach using for instance Faster R-CNN or YOLO architectures will be adopted, as these are now the state-of-the-art for detection problems, and can run in real-time. Nevertheless, this is a worthwhile exercise to better understand traditional machine learning techniques and build intuition on feature selection. Moreover, I was struck by the beauty and simplicity of a technique like HOG, which still manages to produce solid results.
I would like to thank once again my mentor Dylan for his support and advice throughout this term. I am also very grateful to Udacity for putting in place such an exciting and challenging Nanodegree, with great projects and excellent material.
We stand on the shoulders of giants, and therefore I am thankful to all researchers and enthusiasts in the fields of artificial intelligence, computer vision, and beyond, for the work produced and shared via papers and code. Without those resources, I would not have been able to “borrow” their ideas and techniques and successfully complete this project.
Term 1 is now over 🎉🎉. I will begin Term 2 in January 2018, and in the meantime will work on deepening my nascent AI skills as well as resume work on side projects. Thanks for reading and stay tuned!
Thanks for reading this post. I hope you found it useful. I’m now building a new startup called EnVsion! At EnVsion, we’re creating the central repository for UX researchers and product teams to unlock the insights from their user interview videos. And of course we use AI for this ;).
If you’re a UX researcher or product manager feeling overwhelmed with all your video calls with users and customers, then EnVsion is for you!
You also can follow me on Twitter.