Overfitting (What They Are & Train, Validation, Test & Regularization)
By: Chan Woo Kim
Tags: Regularization, Machine Learning, Gradient Descent, Overfitting, Deep Learning

I’ve taken a bit of a hiatus because I’ve been busy with school and other things and I’ve been spending time learning more ML/DL stuff. Here’s another article! We’ll explore what overfitting is, why it is bad, and the solutions that exist to reduce it.

What Is “Overfitting”?

We must first review what a typical machine learning task looks like. I explained in a previous article that, in a typical machine learning task, the researcher divides the dataset. There are two methods people like to go about doing this.

Train & Test

This method is simple: divide the data to a separate training set and a testing set. The training set is where the machine learning algorithm learns from and the testing set is the one used to evaluate the performance of the program. I like to keep a 4:1 ratio, 4/5 of the data dedicated to training the program and 1/5 of the data dedicated to testing the program. The separate testing set should be foreign to the machine learning algorithm and the purpose of it is to test if what it learned can really go out and do useful things accurately(e.g. making predictions on data). The researchers would have to make sure the testing data contains no data that the program has seen previously in the training phase, as otherwise it’d be a skewed test (like giving a test to a student when the student has already seen the test and studied from it).

Train & Validation & Test

This is a similar idea but we have another component: “validation set”. The testing set serves the same purpose: testing the program in the end to gauge its performance as statistics for others to review. Unlike the testing set, the validation set is a dataset that the program will encounter. Here’s the thing: in the training phase, the program could be learning all the wrong things. When we learn concepts, we also make mistakes — we might sometimes extrapolate from what we are studying and convince ourselves about all the incorrect ideas about the world. Programs are prone to do the same thing — recall that all these programs have as teachers are only separate programs that tell it if its guess was right or wrong and forcing it to learn from its mistakes. The validation set comes in to tell the program in the middle of its learning phrase to test if it’s doing the right thing. If it is not, and it keeps getting the predictions wrong in the validation set during the training phase, we know something is going wrong in the learning process and we can fix the necessary hyper-parameters. After its training from the designated training set and the validation set that tests the program and makes it learn from more data, the testing set comes in just like before to test how well the program performs and the program should not learn from its mistakes in the testing set (because that implies even more learning and its performance might be a lot more different after learning from the testing set).

Overfitting

I mentioned a little about mistakes a program can make during the training phase, in analogy to how we learn ourselves. This learning error is largely attributed to what is called “overfitting”. The following image from an Andrew Ng’s lecture makes it clear what overfitting is:

From here.

Forget about the complicated looking math formulas below and just simply look at the graph. Refer back to the article about SVM to view this in context of having decision boundaries; in this example, we want to make it so that if a new unseen data point is on the right of the decision boundary, it will predict to be blue and it will predict to be red if it is on the left.

The given here is that the second graph shows the best decision boundary. Most people should feel that this type of curve represents the best division between blue and red points, albeit small exceptions. Consider the first image. It does a pretty nice job of dividing the dataset but it could be better; a linear line is not the best decision boundary for this particular dataset.

The third graph is what we would say is representative of the program overfitting to the training set. Recall that the primary objective of a machine learning algorithm is to minimize mistakes. In the training phase, the program makes guesses and improves upon its mistakes to better guess in the next data points it receives. So what it will naturally incline towards is the third image. Notice that in the third image, everything on the right of the decision boundary is blue and everything on the left of the decision boundary is red; it technically performs better than the classifier in the middle. But something doesn’t feel right about it.

Generalizing

Your gut feeling is right; the problem is that it does not generalize. We want to train a model that ultimately predicts the category of a future data point. We see that with the learning shown in the third image, this is difficult. We need to accept that datasets have outliers and must strive towards understanding the general patterns that emerge from these data points instead of fitting to every single one of them; if we agree that the second graph is the better model to learn, there are a number of points in the third graph that we would think to classify to be blue but the model in the third graph would not.

Very often, this is the mistake that the validation set wants to test for. During the training phase, even if it is accurately classifying all the data in the training set, if it keeps getting things wrong in the validation set, we can safely assume that it is overfitting to the training set, because that would mean that it is not generalizing well to points it hasn’t encountered.

Real-Life Example

Think of the example of taking tests from before. Imagine a test with only two options, A and B, for its multiple choice questions. We take a different test of this format on and on again and are able to see how we performed every time. Just out of coincidence, it happens to be that odd questions are very likely to have the answer of A and even questions are very likely to be answered by B! We will learn this eventually and do really well as we do more tests and get really good grades. But given a new test without this little cheat to it, we will fail miserably. This would be because we learned all the wrong things from the test; we were supposed to learn the contents in the tests that could generalize to any questions but we instead learned about very specific aspects to the tests given to us and failed to generalize to tests we haven’t seen before. This is analogous to overfitting in the sense that we want to learn a model that can be applied to all data points instead of what is true in our given training set and it is okay to be a bit more incorrect in the training set if that means it will generalize better in the future.

Reducing Overfitting (Regularization)

So how do we reduce these mistakes? One of the most applied ways to do so is using regularization terms and dropout in neural networks. Regularization terms are used in integration with gradient descent.

Gradient Descent

I didn’t explain in the previous article the mathematical formulations of gradient descent but I feel that it is necessary to bring it up here:

Screenshot from my paper but by no means unique or original at all

This looks complicated! Let’s unpack what is going on here. The first half should look familiar to you if you looked into the further exploration section in my previous article. L(WTx + b, yi) refers to the loss function.

The loss function computes the error of our program. This is an operation by which we update the weights ‘w’ based on the error computed by the loss function, and the partial derivatives are signifying how it should update the weights to give a lessor error next time. The weights are essentially what the program is actually deriving when we say a program learns from data. What we want is is this for example:

W = [1,2,3] // weights X = [

[-1,-2,-1],

[4,2,5]

] y_true = [

[0],

[1]

] // y_true is the collection of real categories specified in binary of the datapoint // in the corresponding index in X. W*X[0] = 1*(-1) + 2*(-2) + 3*(-1) = -1-4-3 = -8

W*X[1] = 1*(4) + 2*(2) + 3*(5) = 3+3+15 = 21 If W*X < 0:

predict 0;

Else If W*X>0:

predict 1;

These given weights will predict the data points in X to be the correct classes specified in y_true. Try making the weights different, maybe making them all negative and see what happens (it will be wrong in both cases).

The goal of the training would be to learn the right set of weights ‘w’ that will get us the right answers. Look at the loss function L again: the first parameter is what the model will predict (its weights times the inputted data plus bias) and the second parameter is the actual category of the data points and the function computes how incorrect it was. The program finds those right set of weights by updating its current weights based on how incorrect the program was in its decisions in every step of the process.

This is what we call gradient descent, the intuition of which was covered in the previous article.

Regularization

But what is that R(w) term? That is the regularization term and that is what will let us reduce overfitting. Here’s the idea; look at the graph shown below:

Take a look at this oddly complex function and imagine it being the decision boundary. This bares resemblance to the third image of the image we saw earlier. The outcome of overfitting is usually that it creates an unnecessarily complex function that fits almost every data point. Our objective is to simplify the function to allow it to generalize better. In order to do this, let us first see what the function of the graph is:

Think of the coefficients of the graph as being the weights. Look at the weights example I’ve shown before; you can see the direct relation in terms of how you’re multiplying the coefficient with the x terms.

Now let’s take a look at a more simple graph (colored in blue) that generalizes better:

The function of the function in blue is the following:

Notice that the coefficients are smaller than those from the previous example. The weights are smaller than before. We have penalized the weights, the effect of which is that the influence of the weights are lesser than before. By penalizing the weights, we decreased how specific the model is, allowing it to compute a more simplified graph that probably generalizes better.

This is (kind of) what the regularization term is doing:

Repeated from above

First, to clarify, on the left of the arrow sign is our newly updated set of weights and the ‘w’ on the right of the arrow is our current set of weights that we are about to adjust with gradient descent. The arrow symbolizes an update. Notice that the extra R(w) term makes it so that we subtract MORE from ‘w’ than it would if we didn’t have the R(w) term in there. This makes the new weights smaller than it would be if the regularization term was not there, the same way we get a simpler graph by decreasing the coefficient. The alpha sign in front of the regularization term specifies how big this penalty would be; in some cases the regularization might just make some weights zero completely. This could imply that the function might change to something like this:

The graph of which looks like the following (color in green):

With some coefficients going to zero, the terms from power of 3 to above are completely taken out, which would drastically simplify the function (maybe to a point where the model is under-fitting, where it is not specific enough).

Further Exploration

We see that there is a regularization term but we don’t see the real function that it represents in the equation above. There is a number of functions it could be. In my own research, I used the following:

This is something you can go explore on your own online.