Training Self Driving Cars using Reinforcement Learning | by Jerry Qu | Towards Data Science

By 2040, 95% of new vehicles sold will be fully autonomous.
Wow. 😮
Two reasons why this is revolutionary:
It will save 1.25 MILLION lives every year from traffic accidents
It will give you the equivalence of 3 extra years in a lifetime, currently spent in transit
Self driving cars will become a multi-trillion dollar industry because of this impact. What could you do with 3 extra years? This is the question driving innovation from tech-leaders like Elon Musk and Google. But before we can get there, we need to understand the technology making this all possible, Reinforcement Learning.
Imagine a robot (Also known as an agent) is trying to pick up a pen, and it fails. It tries again, fails. After repeating this process 1000 times, it finally succeeds. The agent has now learned how to pick up a pen. This is Reinforcement Learning in a nutshell, it’s a lot like how living creatures learn.
There are 3 key terms in Reinforcement Learning
State: Describes the current situation. (eg. Position in a maze)
Action: What the agent can do in its situation (eg. Move right)
Reward: Feedback for whether a particular action in a given state was good or bad
In this image, there is a robot at position (1, 1), in a maze. That position is the state. The robot has a set of actions that it can perform, move up or move right. The last thing to note is that, the robot will receive a reward whenever it takes an action. The rewards are defined by the programmer, and we’ll define the rewards as such.
Enter open square: +1
Bump into wall square: -1
Enter exit square: +100
Every time the robot takes an action (a), in state (s), it receives a reward (r). In its head, it constantly updates its memory of rewards for taking certain actions in those states. This is called the policy. The policy is essentially how the agent makes decisions. All decisions are based around predicted rewards. Let’s pretend the robot has a huge table in its head. The table stores all of the states, with all possible actions, and predicted rewards for that state-action pair. At the beginning of training, all predicted rewards are set to 0.
+------------------+--------------------------+| State            | 1, 1                     |+------------------+--------------------------+| Action           | Up | Down | Left | Right |+------------------+----+------+------+-------+| Predicted Reward | 0  | 0    | 0    | 0     |+------------------+----+------+------+-------+
The robot starts by taking random actions, since it has no experience yet. However, after taking random actions over many iterations, it slowly learns to accurately predict rewards for each action. It does this by adjusting its predicted reward for specific state-action pairs towards the received reward every time. These predicted rewards, are formally known as Q-Values. Q-Values essentially define how ‘good’ a certain state is and how much the agent wants to move into that state.
+----------------------------+--------------------------+| State                      | 1, 1                     |+----------------------------+--------------------------+| Action                     | Up | Down | Left | Right |+----------------------------+----+------+------+-------+| Q-Value (Predicted Reward) | 1  | -1   | -1   | 1     |+----------------------------+----+------+------+-------+
Now, the robot can simply look for the highest Q-Value, and take that action. It has learned!
We have two problems with our model right now. One big, and one small. We’ll start with the small one.
Let’s imagine the agent has just started learning so all q-values are zero. The agent is trying to get to the end, meaning it is aiming to choose actions with the highest q-values. Since all start at 0, each has a 25% chance of being picked. Say the agent ends up choosing to move right. The policy now looks like this:
+---------+--------------------------+| State   | 1, 1                     |+---------+--------------------------+| Action  | Up | Down | Left | Right |+---------+----+------+------+-------+| Q-Value | 0  | 0    | 0    | 1     |+---------+----+------+------+-------+
But there’s a problem. The next time the agent resets to state (1, 1), it will always pick the action, right. The agent never had a chance to explore the other actions. This is called a Greedy Policy, where the agent always picks the highest q-value. The agent is “Exploiting” it’s prior knowledge, hence exploitation. The other potential policy is to choose an action randomly, hence exploration. In practice, many factors affect our policy (How we choose which action to take).
Is your agent running in a simulation where it can make mistakes without repercussions? Or is the agent running in real time?
The former would benefit from a higher exploration, to learn faster, while the latter would benefit from a higher exploitation, as to reduce loss. A combination of both exploitation and exploration is normally used, depending on the problem. (Ex. Exploit 70% of the time, and explore 30% of the time)
A bigger problem with our agent right now, is that we can only look one step ahead. For example, at state (1, 1) the agent has learned that both actions will result in a +1 reward. We can look at the maze, and see that moving up would not be an ideal action, since it would lead to a dead end. We need a way to correlate future rewards with the current reward.
The equation above is known as the Future Cumulative Discounted Reward. It is part of the Bellman Equation and it allows us to account for future rewards past state t+1. In this equation:
These are what the variables represent:
(t): The current state
(R t): The cumulative discounted reward at state t (To update our q-value)
(r): The reward
(γ): The discount rate (Between 0 and 1)
(k): The number of steps into the future that the reward was received
For all of you out there who don’t love a bunch of math equations. ✋ This equation essentially takes the sum of the current reward, plus all future rewards, discounted by a number γ that is exponentially decreased by a factor of k (The number of steps into the future the reward was received). Using this allows us to update our q-values based on not only the next action, but all the actions following.
Applying this with a γ = 0.9, we get:
At the final state, there’s only the current reward (100). But for the state before that we take a discount of the next reward plus the current reward (100 * 0.9 + 1 = 91). Then for the state before that we take a larger discount of the final reward plus a discount of the next reward in addition to the current reward (100 * 0.9 * 0.9 + 1 * 0.9 + 1 = 82.9). And so on and so forth. Rewards farther into the future, logically should be discounted greater since the current action has less of a role in generating that reward.
Our agent is now fully equipped to learn! It will explore the maze thousands of times. Each time slightly adjusting its q-values towards R t (The cumulative discounted reward). Only by small amounts because large changes will make the learn too chaotic. After thousands of iterations, the agent will have learned approximate q-values, which it can then utilize with the greedy policy to take the action with the highest q-value, and make it to the end of the maze!
Oh no, not again...
This is the last one, I promise. 😃
Imagine trying to use this form of Reinforcement Learning in a real world scenario. A self driving car for example. The problem we face is that our world is WAY too complex to store q-values for every single state. Imagine storing a value for every single coordinate point on Earth. Now add in other factors like weather conditions, the direction the agent is facing, and an ever changing Earth, and the problem becomes IMPOSSIBLE!
“Persistence is very important. You should not give up unless you are forced to give up.” — Elon Musk
The solution we’ve been using before can be compared to a brute force method where our agent stores q-values for every single state. But what if we used Machine Learning (Neural Networks), to predict q-values for each action, given your state as input. This would completely change the game.
In many real world problems, there are patterns in our states that correspond to q-values. For example, if a self driving car senses a car stopped in front of it, the self driving car must stop! This applies no matter where the self driving car is, what the weather is, or any other factor. The car would crash if it were to continue driving, leading to a VERY negative reward (As defined by the Machine Learning Engineers). So, instead of creating a table of every single possible condition (weather, location, etc), a neural network could simply learn that if a car is stopped in front of the agent, take the action named ‘slow down’.
Neural Networks are amazing at one thing, Learning PATTERNS!
Deep Q-Learning uses Neural Networks to learn the patterns between state and q-value, using the reward as the expected output. These are called Deep Q-Networks.
In a traditional Neural Network, we’d be required to label all of our inputs. That is how neural networks learn. (Check out my last article on Neural Networks if you need a refresher.) However, in the scenario of self driving cars, how are we supposed to define our expected values at every time interval? We’d need to say, turn the wheel 0.5 degrees to the left, increase speed to 50kph. And in a fraction of a second, we’d need new labels for our new input! This would take years to label!
In Deep Q-Networks, instead of defining every single value (Steering wheel angle, speed), we correlate actions to q-values. For example, turn steering wheel 5 degrees to the left is a potential action. So is 4 degrees to the right, and change to 50kph, and change to 30kph. We then define rewards. This is completely up to the engineer, and there is definitely a bit of play here as we try out different values. Crashing is obviously bad, and must be avoided. So, maybe we give it a reward of -1000000. On the other hand, we’ll give the agent a reward of 1 for each second it doesn’t crash, and a reward of 10 for getting closer to its destination. In this method, there is feedback (reward), intrinsic to our environment. We are now simply mapping that feedback, to the actions that caused that feedback. That is the beauty of Reinforcement Learning. We no longer have to label our data!
We aren’t completely done with Reinforcement Learning at this point. There are many advancements being made as we speak! Machine Learning is a new and fluid field. Some of the largest advancements made include:
Experience Replay
Distributional Reinforcement Learning
Separate Target Network (Double Deep Q-Learning)
I’ll quickly skip over these, as they aren’t essential to the understanding of reinforcement learning in general. Experience replay takes the approach of not training our neural network in real time. Instead, it performs many actions, receives a batch of action-state-reward trios, and randomly samples those trios to train the neural network. This allows for a more generalized model as the order of data is randomized.
Distributional Reinforcement Learning is a more recent development where instead of optimizing our neural network to a single q-value for each action, we train it on a distribution of the probabilities of q-value ranges for each action. For example, the outputs of our neural network would become:
Probability of Q-Value of action a being 1–5
Probability of Q-Value of action a being 6–10
Probability of Q-Value of action b being 1–5
Probability of Q-Value of action b being 6–10
This solves problems that arise when q-values have multiple peaks. These multiple peaks would normally be averaged together after numerous iterations, but this new method allows the agent to consider each peak individually. In practice, researchers demonstrated that this model doubled the speed of training for agents playing Atari games!
Yeah. I had to do something with my new found knowledge. Stay tuned to find out how I applied Reinforcement Learning to a virtual self driving car in my next article! (It’s now been published)
Reinforcement Learning, abstracts the need to label data. However, as a result we lose a bit of accuracy. Imagine the difference between being told how to drive a car, versus driving, crashing, and learning on your own.
“It turns out that reinforcement learning is a type of machine learning whose hunger for data is even greater than supervised learning. It is really difficult to get enough data for reinforcement learning algorithms.” — Andrew Ng
Nonetheless, Reinforcement Learning is a stepping stone to a new world. Its drawn us one step closer to General AI, by taking feedback directly from the environment. (Learn more about how I think we could create a General AI). We’ve already proven the value of reinforcement learning in areas such as Machine Trading, and Self Driving Cars. But this is still a new field, constantly being improved and disrupted upon. So stick around. Reinforcement Learning and Machine Learning in general are here to stay!
If you want to take anything away from this article:
Reinforcement Learning abstracts the need to label data in Neural Networks
Deep Q-Learning is a method of Reinforcement Learning
Machine Learning is a dynamic field, constantly being improved and disrupted
If you enjoyed this article:
follow my Medium, LinkedIn, and Twitter to stay updated with my progress!
shoot me an email (Jerry2001Qu@gmail.com) I’ll reply 😉