Deep Learning for Self-Driving Cars | by Manajit Pal | Towards Data Science

Thanks a lot to Valohai for using my rusty tutorial as an intro to their awesome machine learning platform 😍. I would suggest you all to check out their example on how to train the network on the cloud with full version control by using the Valohai machine learning platform (www.valohai.com).
We all know self-driving cars is one of the hottest areas of research and business for the tech giants. What seemed like a science-fiction, a few years ago, now seems more like something which is soon to become a part and parcel of life. The reason, I am saying “soon to be” is because of the fact that even though companies like Tesla, Nissan, Cadillac do have self-driving car assistance software, but, they still require a human to keep an eye on the road and take control when needed. However, it is fascinating to see how far we have come in terms of innovation and how fast technology is advancing. So much so, that now, with the help of basic deep learning, neural network magic, we can build our own pipeline for autonomous driving! Excited? I sure am! Let’s get right into it then...
Prerequisites:1. This article assumes a basic understanding of Convolutional Neural Networks and its working.2. The code mentioned here is written in Python using the Pytorch framework so basic knowledge of the language and the framework is recommended.
If the above seems gibberish to you, do not panic! This free course by Udacity will give you everything you need to know about the basics of Deep Learning and Pytorch.
My Background I started my deep learning journey with one of the Udacity’s scholarship programme sponsored by Facebook through which I learned the basics of Pytorch from the course mentioned above. Simultaneously, I was also enrolled in Udacity’s Self-Driving Car Engineer Nanodegree programme sponsored by KPIT where I got to code an end-to-end deep learning model for a self-driving car in Keras as one of my projects. Therefore, I decided to rewrite the code in Pytorch and share the stuff I learned in this process. Okay, enough of me talking, let’s set up our machine for the awesomeness with one thing in mind — Say NO to overfitting!
Resources for the Project1. Udacity’s self-driving car simulator2. Of course, Python and the Pytorch Framework3. If your machine does not support GPU, then I would recommend using Google Colab to train your network. It provides GPU and TPU hours for free!4. If you are facing problems gathering your training data, you can use the one provided by Udacity to train your network.5. The complete code is available here and the Colab notebook is available here.6. The Nvidia research paper mentioned in this article can be found here.
Gathering DataThe Udacity provided dataset works well but it is not enough to get the car running in difficult terrain (like the second track in Udacity simulator). To gather the data from track 2, we would first need to create a folder in our project directory. Let’s call this folder- data. Now, fire up our simulator. Select the second track from the menu and go to the training mode option.
Once you enter the training mode, you shall see a record option on the top right corner of the screen. Click on the icon and browse to the data folder. Press select.
You can start recording your ride once you press the record icon again! Now, if you are a novice gamer like me, I would suggest to take things slow and try to make sure your car stays at the center of the road as much possible, even during the turns. This would help to get better training data that will eventually make a good model. We will record 2 laps driving in one direction of the track and also 2 more laps driving in the opposite direction to make sure the turns are reversed. This would make sure our model does not overfit and make better left and right turns. Let’s do it!
The training data is now stored in the data folder. Inside, there is one folder and one file: IMG and driving_log.csv. Our next job is to read from the CSV file, the names of the images and their related steering data and load the respective image from the IMG folder in Python.
TIME TO CODE!
It would be better to work with Colab if you prefer not taking the headache of installing different libraries and frameworks in your local machine and also if you would like to take advantage of the free GPU hours. Also, if you prefer not collecting data then you can import the Udacity’s dataset using !wget URL and unzip the file using !unzip. First thing’s first. Importing the headers:
Reading and splitting the data
The code above reads everything from the log file and stores it into the sample array. The line next(reader, None) takes away the first row which contains the names of the columns. We shall use this array to split our data into training and validation. One Good practice from what I have learned is to have 20–30% of training data as the validation set to compare the validation loss and training loss so that we can avoid overfitting. Let’s do that:
Loading Images in DataloaderNow that we have made the samples, it is time to read the images and augment them. This is an important step as this will help to generalize our model. But the process is computationally heavy and time-consuming even for GPUs. The trick is to parallelize this process by taking data in batches, augmenting them and sending to the model to train. Keras achieves this process using python generators and the fit_generator function. In Pytorch, we shall use the Dataset class and the Dataloader function to achieve this. In order to implement this, we would have to overload a few functions of the class, namely, the __getitem__, __len__ and __init__ functions. We would also have to define some augmenting processes. I coded a basic function that will take an image and randomly crop it and flip it horizontally along with taking the negative of the steering data. The cropping, basically, helps the model to focus only on the road by taking away the sky and other distracting stuff in the image and flipping is done to make sure the images are generalized to left and right turns, essentially keeping the car at the center of the road. Other techniques could be adding random brightness to imitate different duration of the day, overlaying the image with a layer of distortion and noise to simulate rain, adding random shadows on the road etc. But we will just stick to the basics for now.
Next up, we define the Dataloader class and pass on this augment function to the input batch samples, concatenate the steering data and images and return it.
Notice that there is also an argument called transform. Our transformation will normalize the image array values to the range 0–1 using a lambda function. Next, we use Dataloader function to add everything in a generator that will be called batch-wise during training. We define a batch size of 32 and shuffle them while passing it to the DataLoader. The num_workers defines how many workers will create the batches in parallel for processing.
Model ArchitectureIt is time to build our model. Let’s look closely at the Nvidia research paper. Go ahead and open it in a new tab. The link is on the resources section above.
If you scroll down to page 5 of the pdf, you will see the architecture of the CNN they built.
Okay, you lazy person, this is the image I am talking about 😝. If you look at the numbers, you shall see the depth of the convolutional layers and the input and output features of the fully-connected layers. Now, every feature maps have some mentioned kernels going over them. As mentioned in the paper, the Nvidia guys used YUV images as input and used strided convolutions in the first three convolutional layers with a 2×2 stride and a 5×5 kernel and a non-strided convolution with a 3×3 kernel size in the last two convolutional layers. Interestingly, however, there was no mention of maxpool layer. I tried to follow the above architecture religiously and build the CNN. Here is what I came up with:
Unfortunately, something was wrong with this approach. Maybe it was a lack of data or maybe the absence of maxpool layers, the network performed horribly. The car was always drifting off road even on a straight track. After some Google search, I came across this repo. The model used here as a simplified version of the Nvidia architecture. I tried and it worked perfectly when trained with enough epochs. So the final architecture for me was this:
However, feel free to try out the first model with maxpool layers. It shall require some calculation for padding and the output height and width. Let’s take a moment to look over a few things here- a) nn.Module- The nn.Module class is used in Pytorch to create a CNN. We have to overload the __init__() and forward() functions to build the network. I made use of nn.Sequential() in order to avoid writing duplicate code. Whatever is in the nn.Sequential() function gets applied to the input sequentially. Pretty neat, eh?a) elu activation- The activation function used here is elu (exponential linear unit). Unlike relu (rectified linear unit), elu speeds up the training process and also solves the vanishing gradient problem. More details and the equation of the elu function can be found here.b) Image Flattening- The flattening of the output from convolutional layers before passing to the fully-connected layers is done with the line: output.view(output.size(0), -1) .
Optimizer and CriterionMoving on, the paper also discusses using Mean Square Error loss as the criterion and an Adam optimizer. Let’s code that out! I set the learning rate to 0.0001. You can adjust it accordingly.
Hmm, we are getting somewhere here. Before we code the final training and validation parts, let’s venture into the world of GPUs!
Pytorch and CUDAPytorch provides an easy integration with CUDA enabled GPUs (sorry AMD). This is done with a simple device() function. It can immensely speed up the training process, like 10x faster than a normal CPU. Why not take advantage of this? For that, we need to transfer our data and the model to GPU for processing. It is really easier than it sounds. We define a function that will do this for all the input it receives.
Notice, that I have converted it to float() so that model is able to compute the inputs.
TrainingTime to train our masterpiece! We first transfer the model to GPU, then, use the generator to get our data and that to the GPU as well. Next, we set the optimizer to not accumulate the gradients during backpropagation using the optimizer.zero_grad() function. Finally, we calculate the total train loss and divide by the batch size to the get the average training loss per epoch. Pretty straightforward.
ValidationSame thing for validation, but this time we will make sure our model is in evaluation mode so that we do not mistakenly update the gradients and backpropagate the errors. We use model.eval() for changing the mode and to make sure the model does not track we use torch.set_grad_enabled(False). The loss is calculated in the same way.
Saving the ModelThe final step of the code is here! After the training is complete, we save the model to use it for driving the car autonomously in our simulator. We make a state dictionary and save the model with .h5 format using torch.save().
The Fun Stuff BeginsOne final thing before we begin testing our model, we need a file that will load our model, get the frames of the track from the simulator to process through our model and send the steering prediction back to the simulator. Fear not! I have made a drive.py file which basically a Pytorch version of Udacity’s drive.py file I used in my project. You can go through the code and experiment with it if you want different throttle etc. For now, let’s get copy-paste the content of the below code.
github.com
Also, we would need a model.py file which shall contain the model architecture. Create the file and paste your network architecture. If you face any problem, feel free to take a look at my model.py file in the repo.
Download the model.h5 file in the same directory if you had been using Google Colab to write your code. Fire up the terminal, cd to your directory and run the script with our model:
python drive.py model.h5 
If you have two different python version installed in your machine, use python3 instead of python. Click on allow, when you see any popups.
Open the simulator again and now choose the autonomous mode. The car should drive on its own like a boss!
There you go! Your very own self-driving car pipeline. How cool is that! 😁
Here is a little demo of what you might expect:
TroubleshootingYou might see the car wobble a lot or maybe it is confined to one side of the road. This might mean that the data is not augmented and generalized properly. Try to acquire more training data or augment with more randomness.
You might also see the car not moving at all. In that case, check the terminal for any errors during the run. Most of the errors should be solved by installing the appropriate dependency of the library.
ConclusionThanks a lot for taking the time to read my article. I really hope that this helps someone who wants to learn some concepts about the use of Deep learning in self-driving cars. Also, this is my first article related to AI and I am a total beginner so I would really appreciate if you could take some to leave any positive or negative feedback, your thoughts or other resources that you might think is a better way to build AI for self-driving cars in the comments below. Cheers folks!