AWS DeepRacer, Reinforcement Learning 101, and a small lesson in AI Governance
By: Abhimanyu Basu
Tags: Reinforcement Learning, Machine Learning, Aws Deepracer, Ai Governance

Dear readers, hope you are all doing well. I recently participated in an AWS DeepRacer tournament held by my organization (Deloitte) and today I am going to share my experience related to this fun and exciting event. For those who are unfamiliar with AWS DeepRacer, it is a virtual autonomous racing league where participants need to train a vehicle bot using reinforcement learning. The trained bots are deployed on a specific racing track and the bot with the lowest lap time wins. The bots are trained exclusively on AWS infrastructure using managed services like AWS SageMaker and AWS RoboMaker. Thus the DeepRacer service provides a fun and engaging way to get your hands dirty on concepts of reinforcement learning without getting bogged down with too much technicalities. Throughout this article, I will embed links to blogs which I came across during my research to explain related concepts. I found the blogs to be quite nicely written and they can serve as a good refresher to the experienced data scientist as well. However, feel free to skip them if you are already well-versed with these concepts. So without further ado, let’s dive right in.

First, let me show you my final model and how it performed. I will be referring to this video throughout my post so that it is easily relatable.

As you can see, the bot gets three trials to complete the lap. To be considered for evaluation, a trained bot needs to complete the lap atleast one out of three times. My bot completed the lap in the very first run while it was unable to do so in the second and third attempts. I achieved a best laptime of 12.945 seconds which gave me a position of #23 out of 58 participants, while the laptime of the bot that held the #1 position was 10.748 seconds i.e., a difference of +2.197 seconds only!!! So although this was a community race and not a professional competition, the level of competition was fierce and the best lap times achieved by the top 50 percentile were very much in line with actual competitions.

This brings to my next question — how to build such a model? I went from zero to full deployment in literally 2 days, so this is definitely not rocket science. So let’s take it from the start.

Firstly, the entire DeepRacer competition is based on reinforcement learning. Details of reinforcement learning can be found here and here. Simply put, a reinforcement learning is a machine learning model that trains an agent (in this case, the vehicle bot) to undertake complex behaviors without any labeled training data by making it take short term actions (in this case, driving around the circuit) while optimizing for a long term goal (in this case, minimizing lap completion time). This differs significantly from supervised learning techniques. In fact, I found the process of reinforcement learning to be quite similar to how humans learn through trial and error.

This brings to my second point — how does the model learn? Well, at the heart of every reinforcement learning model is something called as a neural network. A neuron takes one or more inputs and produces an output. It does this using an activation function (more about this here). However a single neuron is limited — it cannot perform complex logical operations alone. For complex operations, we use multiple layers of neurons (called as neural networks). A 3-layer network is easier to train than a 5-layer network but the latter will be able to perform more complex optimizations due to the additional layers. DeepRacer allows you to choose between 3-layer CNNs (convoluted neural networks) and 5-layer CNNs and I experimented with both. However, given that I had just 48 hours to train my model, I observed my 3-layer CNN to perform better than my 5-layer CNN. This is how my vehicle bot looked like below. It had a max speed of 2.8m/s, max steering angle of 30 degrees and only 1 camera as a sensor. These parameters are critical to how the bot performs and needs to be chosen through trial and error.