Introduction to Deep Learning for Self Driving Cars (Part — 1)
By: Prateek Sawhney
Tags: Machine Learning, Self Driving Cars, Artificial Intelligence, Deep Learning, Data Science

Measuring Performance

After we have trained our first model, there is something very important to discuss. We have a training set, as well as a validation set, and a test set. What is that all about? Don’t skip that part. It has to do with measuring how well we’re doing without accidentally shooting ourself in the foot, and it is a lot more subtle then we might initially think. It’s also very important because

as we will discover later, once we know how to measure our performance on a problem, we’ve already solved half of it. Let me explain why measuring

performance is subtle.

Let’s go back to our classification task. We’ve got a whole lot of images with labels. We could say, okay, I’m going to run my classifier on those images, and see how many I got right. That’s my error measure. And then we go out and

use your classifier on new images, images that we’ve never seen in the past, and we measure how many we get right, and our performance gets worse. The classifier doesn’t do as well. So what happened?

Well, imagine I construct a classifier that simply compares the new image to any of the other images that I’ve already seen in my training set, and just returns the label. By the measure we defined earlier, it’s a great classifier. It would get 100% accuracy on the training set. But as soon as it sees a new image, it’s lost. It has no idea what to do. It’s not a great classifier. The problem is that our classifier has memorized the training set, and it fails to generalize to new examples. It’s not just a theoretical problem. Every classifier that we will build will tend to try and memorize the training set. And it will usually do that very, very well.

Our job though, is to help it generalize to new data instead. So, how do we measure generalization instead of measuring how well the classifier

memorized the data?

The simplest way is to take a small subset of the training set, not use it in training, and measure the error on that test data. Problem solved, now our classifier cannot cheat because it never sees the test data, so it can’t memorize it. But there is still a problem, because training a classifier is usually a process of trial and error. We try a classifier, we measure its performance and then we try another one and we measure again. And another, and another, we tweak the model, we explore the parameters, we measure, and finally, we have what we think is the perfect classifier. And then after all this care we’ve taken to separate our test data from our training data and only measuring our performance on the test data, now we deploy our system in a real production environment.

Image by Mika Baumeister on Unsplash

And we get more data and we score our performance on that new data and it doesn’t do nearly as well. What can possibly have happened? What happened is that our classifier has seen our test data, indirectly, through our own eyes. Every time we made a decision about which classifier to use, which parameter to tune, we actually give information to our classifier about the test set. Just a tiny bit, but it adds up. So over time, as we run many, and many experiments, our test data bleeds into our training data. So what can you do?