Introduction to Neural Networks For Self Driving Cars (Foundational Concepts Part — 1)
By: Prateek Sawhney
Tags: Machine Learning, Data Science, Neural Networks, Deep Learning, Self Driving Cars

Softmax Function

Let’s say we have a machine learning model that will predict if we receive a gift or not. So, the model use predictions in the following way. It says, the probability that we get a gift is 0.8, which automatically implies that the probability that we don’t receive a gift is 0.2.

And what does the model do?

What the model does is take some inputs. For example, is it your birthday or have it been good all year? And based on those inputs, the ML model calculates a linear model which would be the score. Then, the probability that we get the gift or not is simply the sigmoid function applied to that score generated by the model. Now, what if you had more options than just getting a gift or not a gift? Let’s say we have a model that just tell us what animal we just saw, and the options are a duck, a beaver and a walrus. We want a machine learning model that tells an answer along the lines of, the probability of a duck is 0.67, the probability of a beaver is 0.24, and the probability of a walrus is 0.09. Notice that the probabilities need to add to one. Let’s say we have a linear model based on some inputs. The inputs could be, does it have a beak or not? Number of teeth. Number of feathers. Hair, no hair. Does it live in the water? Does it fly? We will a calculate linear function based on those inputs, and let’s say we get some scores. So, the duck gets a score of two, and the beaver gets a score of one, walrus gets a score of zero. And now the question is, how do we turn these scores into probabilities?

The first thing we need to satisfy with probabilities is as we said, they need to add to one. So the two, the one, and the zero do not add to one. The second thing we need to satisfy is, since the duck had a higher score than the beaver and the beaver had a higher score than the walrus, then we want the probability of the duck to be higher than the probability of the beaver, and the probability of the beaver to be higher than the probability of the walrus.

Here’s a simple way of doing it. Let’s take each score and divide it by the sum of all the scores. The two becomes two divided (2+1+0), the one becomes one divided by (2+1+0), and the zero becomes zero divided (2+1+0). This kind of works because the probabilities we obtain are two thirds for the duck, one third for the beaver, and zero for the walrus. That works but there’s a little problem. Let’s think about it. What could this problem be?

The problem is the following.

What happens if our scores are negative? This is completely possible since the scores are linear function which could give negative values. What if we had, say, scores of 1, 0 and (-1)? Then, one of the probabilities would turn into one divided by (1+0+(-1)) which is zero, and we know very well that we cannot divide by zero. This unfortunately won’t work, but the idea is good.

How can we turn this idea into one that works all the time even for negative numbers? Well, it’s almost like we need to turn these scores into positive scores. How do we do this? Is there a function that can help us?

The answer is Softmax.

The softmax function is a function that turns a vector of K real values into a vector of K real values that sum equal to 1. The input values can be positive, zero, negative, or greater than one, but the softmax transforms them into values between 0 and 1, so that they can be interpreted as probabilities 😀