Title: Understanding Regularization for Image Classification and Machine Learning - PyImageSearch

URL Source: https://pyimagesearch.com/2016/09/19/understanding-regularization-for-image-classification-and-machine-learning/

Published Time: 2016-09-19T14:00:06+00:00

Markdown Content:
> _Many strategies used in machine learning are explicitly designed to reduce the test error, possibly at the expense of increased training error. These strategies are collectively known as regularization._
> 
> _—_[Goodfellow, Bengio, and Courville (2016)](https://www.deeplearningbook.org/)

In previous lessons, we discussed two important loss functions: Multi-class SVM loss and cross-entropy loss. We then discussed gradient descent and how a network can actually learn by updating the weight parameters of a model. While our loss function allows us to determine how well (or poorly) our set of parameters are performing on a given classification task, the loss function itself does not take into account how the weight matrix “looks.”

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20281'%3E%3C/svg%3E)

What do I mean by “looks”? Well, keep in mind that we are working in a real-valued space, thus there are an _infinite set_ of parameters that will obtain reasonable classification accuracy on our dataset (for some definition of “reasonable”).

How do we go about choosing a set of parameters that help ensure our model generalizes well? Or, at the very least, lessen the effects of overfitting. **_The answer is regularization._** Second only to your learning rate, regularization is the most important parameter of your model that you can tune.

There are various types of regularization techniques, such as L1 regularization, L2 regularization (commonly called “weight decay”), and [Elastic Net](https://doi.org/10.1111/j.1467-9868.2005.00503.x), that are used by updating the loss function itself, adding an additional parameter to constrain the capacity of the model.

We also have types of regularization that can be _explicitly_ added to the network architecture — dropout is the quintessential example of such regularization. We then have _implicit_ forms of regularization that are applied during the training process. Examples of implicit regularization include data augmentation and early stopping.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

### **What Is Regularization and Why Do We Need It?**

**Regularization helps us control our model capacity**, ensuring that our models are better at making (correct) classifications on data points that they were _not_ trained on, which we call **_the ability to generalize_**. If we don’t apply regularization, our classifiers can easily become too complex and _overfit_ to our training data, in which case we lose the ability to generalize to our testing data (and data points _outside_ the testing set as well, such as new images in the wild).

However, too much regularization can be a bad thing. We can run the risk of _underfitting_, in which case our model performs poorly on the training data and is not able to model the relationship between the input data and output class labels (because we limited model capacity too much). For example, consider the following plot of points, along with various functions that fit to these points (**Figure 1**).

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20354%20341'%3E%3C/svg%3E)

**Figure 1:** An example of underfitting (orange line), overfitting (blue line), and generalizing (green line). Our goal when building deep learning classifiers is to obtain these types of “green functions” that fit our training data nicely, but avoid overfitting. Regularization can help us obtain this type of desired fit.

The orange line is an example of _underfitting_ — we are not capturing the relationship between the points. On the other hand, the blue line is an example of _overfitting_ — we have too many parameters in our model, and while it hits all points in the dataset, it also wildly varies between the points. It is not a smooth, simple fit that we would prefer. We then have the green function which also hits all points in our dataset, but does so in a much more predictable, simple manner.

The goal of regularization is to obtain these types of “green functions” that fit our training data nicely, but avoid overfitting to our training data (blue) or failing to model the underlying relationship (orange). Regularization is a critical aspect of machine learning and we use regularization to control model generalization. To understand regularization and the impact it has on our loss function and weight update rule, let’s proceed to the next lesson.

### **Updating Our Loss and Weight Update to Include Regularization**

Let’s start with our cross-entropy loss function:

**(1)** ![Image 4: L_{i} = -\log\left(\displaystyle\frac{e^{s_{y_{i}}}}{\sum\limits_{j} e^{s_{j}}}\right)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20144%2071'%3E%3C/svg%3E)

The loss over the entire training set can be written as:

**(2)** ![Image 5: L = \displaystyle\frac{1}{N} \sum\limits^{N}_{i=1} L_{i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2095%2049'%3E%3C/svg%3E)

Now, let’s say that we have obtained a weight matrix **_W_** such that _every data point_ in our training set is classified correctly, which means that our loss _L_ \= 0 for all _L__i_.

Awesome, we’re getting 100% accuracy — but let me ask you a question about this weight matrix — **_is it unique?_** **Or, in other words, are there** **_better_** **choices of** **_W_** **that will improve our model’s ability to generalize and reduce overfitting?**

If there is such a **_W_**, how do we know? And how can we incorporate this type of penalty into our loss function? The answer is to define a **regularization penalty**, a function that operates on our weight matrix. The regularization penalty is commonly written as a function, _R_(**_W_**). Equation (**3)** shows the most common regularization penalty, L2 regularization (also called **weight** **decay**):

**(3)** ![Image 6: R(W) = \sum\limits_{i}\sum\limits_{j} W_{i,j}^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20132%2031'%3E%3C/svg%3E)

What is the function doing exactly? In terms of Python code, it’s simply taking the sum of squares over an array:

penalty = 0

for i in np.arange(0, W.shape\[0\]):
	for j in np.arange(0, W.shape\[1\]):
		penalty += (W\[i\]\[j\] \*\* 2)

What we are doing here is looping over all entries in the matrix and taking the sum of squares. The sum of squares in the L2 regularization penalty discourages large weights in our matrix **_W_**, preferring smaller ones. Why might we want to discourage large weight values? In short, by penalizing large weights, we can improve the ability to generalize, and thereby reduce overfitting.

Think of it this way — the larger a weight value is, the more influence it has on the output prediction. Dimensions with larger weight values can almost singlehandedly control the output prediction of the classifier (provided the weight value is large enough, of course) which will almost certainly lead to overfitting.

To mitigate the effect various dimensions have on our output classifications, we apply regularization, thereby seeking **_W_** values that take into account _all_ of the dimensions rather than the few with large values. In practice you may find that regularization hurts your training accuracy slightly, but actually _increases your testing accuracy_.

Again, our loss function has the same basic form, only now we add in regularization:

**(4)** ![Image 7: L = \displaystyle\frac{1}{N} \sum\limits^{N}_{i=1} L_{i} + \lambda R(W)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20167%2049'%3E%3C/svg%3E)

The first term we have seen before — it is the average loss over all samples in our training set.

**The second term is new —** **_this is our regularization penalty_****.** The _λ_ variable is a hyperparameter that controls the _amount_ or _strength_ of the regularization we are applying. In practice, both the learning rate _α_ and the regularization term _λ_ are the hyperparameters that you’ll spend the most time tuning.

Expanding cross-entropy loss to include L2 regularization yields the following equation:

**(5)** ![Image 8: L = \displaystyle\frac{1}{N} \sum\limits^{N}_{i=1}\left [-\log\left(\displaystyle\frac{e^{s_{y_{i}}}}{\sum\limits_{j} e^{s_{j}}}\right)\right] +\lambda \sum\limits_{i} \sum\limits_{j} W_{i, j}^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20327%2071'%3E%3C/svg%3E)

We can also expand Multi-class SVM loss as well:

**(6)** ![Image 9: L = \displaystyle\frac{1}{N} \sum\limits^{N}_{i=1} \sum\limits_{j \neq y_{i}} \left[\max\left(0, s_{j} - s_{y_{i}} + 1\right)\right] + \lambda \sum\limits_{i} \sum\limits_{j} W_{i, j}^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20372%2051'%3E%3C/svg%3E)

Now, let’s take a look at our standard weight update rule:

**(7)** _W_ \= _W −α_∇**_W_** _f_(_W_)

This method updates our weights based on the gradient multiplied by a learning rate _α_. Taking into account regularization, the weight update rule becomes:

**(8)** _W_ \= _W −α_∇**_W_** _f_(_W_)_−λR_(_W_)

Here, we are adding a negative linear term to our gradients (i.e., gradient descent), penalizing large weights, with the end goal of making it easier for our model to generalize.

### **Types of Regularization Techniques**

In general, you’ll see three common types of regularization that are applied directly to the loss function. The first, we reviewed earlier, L2 regularization (aka “weight decay”):

**(9)** ![Image 10:  R(W) = \sum\limits_{i}\sum\limits_{j} W_{i,j}^{2}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20132%2031'%3E%3C/svg%3E)

We also have L1 regularization which takes the absolute value rather than the square:

**(10)** ![Image 11:  R(W) = \sum\limits_{i}\sum\limits_{j} |W_{i,j}|](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20142%2031'%3E%3C/svg%3E)

[Elastic Net](https://doi.org/10.1111/j.1467-9868.2005.00503.x) regularization seeks to combine both L1 and L2 regularization:

**(11)** ![Image 12: R(W) = \sum\limits_{i}\sum\limits_{j} \beta W_{i,j}^{2} + \left|W_{i,j}\right|](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20200%2031'%3E%3C/svg%3E)

Other types of regularization methods exist such as directly modifying the architecture of a network along with how the network is actually trained — we will review these methods in later lessons.

In terms of _which_ regularization method you should be using (including none at all), you should treat this choice as a hyperparameter you need to optimize over and perform experiments to determine _if_ regularization should be applied, and if so _which method_ of regularization, and what the proper value of _λ_ is. For more details on regularization, refer to [Chapter 7 of Goodfellow et al. (2016)](https://www.deeplearningbook.org/), the [“Regularization” section from the DeepLearning.net tutorial](https://www.deeplearningbook.org/slides/07_regularization.pdf), and the [notes from Karpathy’s cs231n Neural Networks II lecture](https://cs231n.github.io/neural-networks-2/).

### **Regularization Applied to Image Classification**

To demonstrate regularization in action, let’s write some Python code to apply it to our “Animals” dataset. Open a new file, name it `regularization.py`, and insert the following code:

\# import the necessary packages
from sklearn.linear\_model import SGDClassifier
from sklearn.preprocessing import LabelEncoder
from sklearn.model\_selection import train\_test\_split
from pyimagesearch.preprocessing import SimplePreprocessor
from pyimagesearch.datasets import SimpleDatasetLoader
from imutils import paths
import argparse

**Lines 2-8** import our required Python packages. We’ve seen all of these imports before, except the scikit-learn `SGDClassifier`. As the name of this class suggests, this implementation encapsulates all the concepts we have reviewed in our lessons, including:

*   Loss functions
*   Number of epochs
*   Learning rate
*   Regularization terms

Thus making it the perfect example to demonstrate all these concepts in action.

Next, we can parse our command line arguments and grab the list of images from disk:

\# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-d", "--dataset", required=True,
	help="path to input dataset")
args = vars(ap.parse\_args())
	
# grab the list of image paths
print("\[INFO\] loading images...")
imagePaths = list(paths.list\_images(args\["dataset"\]))

Given the image paths, we’ll resize them to 32_×_32 pixels, load them from disk into memory, and then flatten them into a 3,072-dim array:

\# initialize the image preprocessor, load the dataset from disk,
# and reshape the data matrix
sp = SimplePreprocessor(32, 32)
sdl = SimpleDatasetLoader(preprocessors=\[sp\])
(data, labels) = sdl.load(imagePaths, verbose=500)
data = data.reshape((data.shape\[0\], 3072))

We’ll also encode the labels as integers and perform a training testing split, using 75% of the data for training and the remaining 25% for testing:

\# encode the labels as integers
le = LabelEncoder()
labels = le.fit\_transform(labels)

# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
(trainX, testX, trainY, testY) = train\_test\_split(data, labels,
	test\_size=0.25, random\_state=42)

Let’s apply a few different types of regularization when training our `SGDClassifier`:

\# loop over our set of regularizers
for r in (None, "l1", "l2"):
	# train a SGD classifier using a softmax loss function and the
	# specified regularization function for 10 epochs
	print("\[INFO\] training model with '{}' penalty".format(r))
	model = SGDClassifier(loss="log", penalty=r, max\_iter=10,
		learning\_rate="constant", tol=1e-3, eta0=0.01, random\_state=12)
	model.fit(trainX, trainY)

	# evaluate the classifier
	acc = model.score(testX, testY)
	print("\[INFO\] {} penalty accuracy: {:.2f}%".format(r,
		acc \* 100))

**Line 37** loops over our regularizers, including no regularization. We then initialize and train the `SGDClassifier` on **Lines 41-43**.

We’ll be using cross-entropy loss, with regularization penalty of `r` and a default _λ_ of 0.0001. We’ll use SGD to train the model for 10 epochs with a learning rate of _α_ \= 0_._01\. We then evaluate the classifier and display the accuracy results to our screen on **Lines 46-48**.

To see our SGD model trained with various regularization types, just execute the following command:

$ python regularization.py --dataset dataset/animals
\[INFO\] loading images...
...
\[INFO\] training model with 'None' penalty
\[INFO\] 'None' penalty accuracy: 50.13%
\[INFO\] training model with 'l1' penalty
\[INFO\] 'l1' penalty accuracy: 52.67%
\[INFO\] training model with 'l2' penalty
\[INFO\] 'l2' penalty accuracy: 57.20%

We can see with _no regularization_ we obtain an accuracy of **50.13%**. Using L1 regularization our accuracy increases to **52.67%**. L2 regularization obtains the highest accuracy of **57.20%**.

**_Remark:_** Using different `random_state` values for `train_test_split` will yield different results. The dataset here is too small and the classifier too simplistic to see the full impact of regularization, so consider this a “worked example.” As we continue to work through this book you’ll see more advanced uses of regularization that will have dramatic impacts on your accuracy.

Realistically, this example is too small to show all the advantages of applying regularization — for that, we’ll have to wait until we start training Convolutional Neural Networks. However, in the meantime simply appreciate that regularization can provide a boost in our testing accuracy and reduce overfitting, _provided we can tune the hyperparameters right_.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In these lessons, we popped the hood on deep learning and took a deep dive into the engine that powers modern day neural networks — _gradient descent_. We investigated two types of gradient descent:

1.  The standard vanilla flavor.
2.  The stochastic version that is more commonly used.

Vanilla gradient descent performs only _one_ weight update per epoch, making it very slow (if not impossible) to converge on large datasets. The stochastic version instead applies _multiple_ weight updates per epoch by computing the gradient on small mini-batches. By using SGD we can dramatically reduce the time it takes to train a model while also enjoying lower loss and higher accuracy. Typical batch sizes include 32, 64, 128, and 256.

Gradient descent algorithms are controlled via a **_learning rate_**: this is by far the most important parameter to tune correctly when training your own models.

If your learning rate is too large, you’ll simply bounce around the loss landscape and not actually “learn” any patterns from your data. On the other hand, if your learning rate is too small, it will take a prohibitive number of iterations to reach even a reasonable loss. To get it just right, you’ll want to spend the majority of your time tuning the learning rate.

We then discussed **_regularization_**, which is defined as _“any method that increases testing accuracy perhaps at the expense of training accuracy.”_ Regularization encompasses a broad range of techniques. We specifically focused on regularization methods that are applied to our loss functions and weight update rules, including L1 regularization, L2 regularization, and Elastic Net.

In terms of deep learning and neural networks, you’ll commonly see L2 regularization used for image classification — the trick is tuning the _λ_ parameter to include just the right amount of regularization.

At this point, we have a sound foundation of machine learning, but we have yet to investigate neural networks or train a custom neural network from scratch. That will all change in a future lesson, where we discuss neural networks, the backpropagation algorithm, and how to train your own neural networks on custom datasets.

![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!