Title: Using GridSearchCV for hyperparameter tuning with scikit-learn - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/24/grid-search-hyperparameter-tuning-with-scikit-learn-gridsearchcv/

Published Time: 2021-05-24T14:00:00+00:00

Markdown Content:
In this tutorial, you will learn how to use the `GridSearchCV` class to do grid search hyperparameter tuning using the scikit-learn machine learning library. We’ll apply the grid search to a computer vision project.

![Image 1: GridSearchCV, SciKitLearn, OpenCV](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)

This blog post is part two in our four-part series on hyperparameter tuning:

1.  [_Introduction to hyperparameter tuning with scikit-learn and Python_](https://pyimagesearch.com/2021/05/17/introduction-to-hyperparameter-tuning-with-scikit-learn-and-python/) (last week’s tutorial)
2.  _Grid search hyperparameter tuning with scikit-learn’s GridSearchCV class_ (today’s post)
3.  _Hyperparameter tuning for Deep Learning with scikit-learn, Keras, and TensorFlow_ (next week’s post)
4.  _Easy Hyperparameter Tuning with Keras Tuner and TensorFlow_ (tutorial two weeks from now)

Last week we learned how to tune hyperparameters to a Support Vector Machine (SVM) trained to predict the age of a marine snail. This was a good introduction to the concept of hyperparameter tuning, but it didn’t demonstrate how to apply hyperparameter tuning to a computer vision project.

**Today, we’ll build a computer vision system to automatically recognize the _texture_ of an object in an image. We will use hyperparameter tuning to find the optimal set of hyperparameters that yields the highest accuracy.**

You can use the code included with this post as a starting point when you need to tune hyperparameters in your own projects.

**To learn how to grid search hyperparameters with `GridSearchCV` and scikit-learn, _just keep reading._**

In the first part of this tutorial, we’ll discuss:

1.  What a grid search is
2.  How a grid search can be applied to hyperparameter tuning
3.  How the scikit-learn machine learning library implements grid search through the `GridSearchCV` class

From there, we’ll configure our development environment and review our project directory structure.

I’ll then show you how to use computer vision, machine learning, and grid search hyperparameter tuning to tune the parameters to a texture recognition pipeline, resulting in a system with near 100% texture recognition accuracy.

By the end of this guide, you’ll have a strong understanding of how to apply a grid search to the hyperparameters of a computer vision project.

### **What is a hyperparameter grid search?**

![Image 2:  Grid search hyperparameter tuning](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20588%20485'%3E%3C/svg%3E)

**Figure 1:** Hyperparameter tuning using a grid search ([image source](https://maelfabien.github.io/machinelearning/Explorium4/)).

A grid search allows us to exhaustively test all possible hyperparameter configurations that we are interested in tuning.

Later in this tutorial, we’ll tune the hyperparameters of a Support Vector Machine (SVM) to obtain high accuracy. The hyperparameters to an SVM include:

1.  **Kernel choice:** linear, polynomial, radial basis function
2.  **Strictness (`C`):** Typical values are in the range of `0.0001` to `1000`
3.  **Kernel-specific parameters:** degree (for polynomial) and gamma (RBF)

For example, consider the following list of possible hyperparameters:

parameters = \[
	{"kernel":
		\["linear"\],
		"C": \[0.0001, 0.001, 0.1, 1, 10, 100, 1000\]},
	{"kernel":
		\["poly"\],
		"degree": \[2, 3, 4\],
		"C": \[0.0001, 0.001, 0.1, 1, 10, 100, 1000\]},
	{"kernel":
		\["rbf"\],
		"gamma": \["auto", "scale"\],
		"C": \[0.0001, 0.001, 0.1, 1, 10, 100, 1000\]}
\]

**A grid search will** **_exhaustively_** **test all possible combinations of these hyperparameters, training an SVM for each set.** The grid search will then report the best hyperparameters (i.e., the ones that maximized accuracy).

### **Configuring your development environment**

To follow this guide, you need to have the following libraries installed on your machine:

*   [OpenCV](https://github.com/opencv/opencv)
*   [scikit-learn](https://scikit-learn.org/stable/)
*   [scikit-image](https://scikit-image.org/)
*   [imutils](https://github.com/jrosebr1/imutils)

Luckily, both of these packages are pip-installable:

$ pip install opencv-contrib-python
$ pip install scikit-learn
$ pip install scikit-image
$ pip install imutils

**If you need help configuring your development environment for OpenCV, I _highly recommend_ that you read my** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in a matter of minutes.

### **Having problems configuring your development environment?**

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 2:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux systems?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

### **Our example texture dataset**

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20205'%3E%3C/svg%3E)

**Figure 3:** Our example texture image dataset includes three classes: brick, marble, and sand.

We’ll create a computer vision and machine learning model capable of automatically recognizing the texture of an object in an image.

**There are three textures we’ll train our model to recognize:**

1.  Brick
2.  Marble
3.  Sand

Each class has 30 images each for a total of 90 images in the dataset.

Our goal is to now:

1.  Quantify the texture of each image in the dataset
2.  Define the set of hyperparameters we’re going to search over
3.  Use a grid search to tune the hyperparameters and find the values that maximize our texture recognition accuracy

**_Note:_** _This dataset was created by following my tutorial on_ [_creating an image dataset with Google Images._](https://pyimagesearch.com/2017/12/04/how-to-create-a-deep-learning-dataset-using-google-images/) _I’ve provided the example texture dataset inside the_ **_“Downloads”_** _associated with this tutorial. That way, you don’t have to recreate the dataset yourself._

### **Project structure**

Before we can implement a grid search for hyperparameter tuning, let’s take a second to review our project directory structure.

Start with the **_“Downloads”_** section of this tutorial to access the source code and example texture dataset.

From there, unzip the archive, and you should find the following project directory:

$ tree . --dirsfirst --filelimit 10
.
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   └── localbinarypatterns.py
├── texture\_dataset
│   ├── brick \[30 entries exceeds filelimit, not opening dir\]
│   ├── marble \[30 entries exceeds filelimit, not opening dir\]
│   └── sand \[30 entries exceeds filelimit, not opening dir\]
└── train\_model.py

5 directories, 3 files

The `texture_dataset` contains the dataset where we’ll train our model. We have three subdirectories, `brick`, `marble,` and `sand`, each with 30 images.

We’ll use [Local Binary Patterns (LBPs)](https://pyimagesearch.com/2015/12/07/local-binary-patterns-with-python-opencv/) to quantify the contents of each image in the texture dataset. The LBP image descriptor is implemented inside the `localbinarypatterns.py` file inside the `pyimagesearch` module.

The `train_model.py` script is responsible for:

1.  Loading all images in `texture_dataset` from disk
2.  Quantifying each of the images using LBPs
3.  Performing a grid search on the hyperparameter space to determine the values that optimize accuracy

Let’s get started implementing our Python scripts.

### **Our Local Binary Pattern (LBP) descriptor**

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201248%20364'%3E%3C/svg%3E)

**Figure 4:** We’ll use the Local Binary Patterns descriptor to quantify the texture of our input image. (Thanks to Bikramjot of Hanzra Tech for the inspiration on this visualization!)

The Local Binary Patterns implementation we’ll follow today [comes from my previous tutorial.](https://pyimagesearch.com/2015/12/07/local-binary-patterns-with-python-opencv/) While I’ve included the full code here as a matter of completeness, I will defer a detailed review of the implementation to my previous blog post.

With that said, open the `localbinarypatterns.py` file in the `pyimagesearch` module of your project directory structure, and we can get started:

\# import the necessary packages
from skimage import feature
import numpy as np

class LocalBinaryPatterns:
	def \_\_init\_\_(self, numPoints, radius):
		# store the number of points and radius
		self.numPoints = numPoints
		self.radius = radius

**Lines 2 and 3** import our required Python packages. The `feature` submodule of scikit-image contains the `local_binary_pattern` function — this method computes the LBPs from an input image.

Next, we define our `describe` function:

	def describe(self, image, eps=1e-7):
		# compute the Local Binary Pattern representation
		# of the image, and then use the LBP representation
		# to build the histogram of patterns
		lbp = feature.local\_binary\_pattern(image, self.numPoints,
			self.radius, method="uniform")
		(hist, \_) = np.histogram(lbp.ravel(),
			bins=np.arange(0, self.numPoints + 3),
			range=(0, self.numPoints + 2))

		# normalize the histogram
		hist = hist.astype("float")
		hist /= (hist.sum() + eps)

		# return the histogram of Local Binary Patterns
		return hist

This method accepts an input `image` (i.e., the image we want to compute LBPs for) along with a small epsilon value. As we’ll see, the `eps` value prevents division by zero errors when normalizing the resulting LBP histogram to the range _\[0, 1\]._

From there, **Lines 15 and 16** compute the uniform LBPs from the input image. Given the LBPs, we then use NumPy to construct a histogram of each LBP type (**Lines 17-19**).

The resulting histogram is then scaled to the range _\[0, 1\]_ (**Lines 22 and 23**).

For a more detailed review of our LBP implementation, be sure to refer to my tutorial, [_Local Binary Patterns with Python & OpenCV._](https://pyimagesearch.com/2015/12/07/local-binary-patterns-with-python-opencv/)

### **Implementing our grid search for hyperparameter tuning using gridsearchcv**.

With our LBP image descriptor implemented, we can create our grid search hyperparameter tuning script.

Open the `train_model.py` file in your project directory, and we’ll get started:

\# import the necessary packages
from pyimagesearch.localbinarypatterns import LocalBinaryPatterns
from sklearn.model\_selection import GridSearchCV
from sklearn.metrics import classification\_report
from sklearn.svm import SVC
from sklearn.model\_selection import train\_test\_split
from imutils import paths
import argparse
import time
import cv2
import os

**Lines 2-11** import our required Python packages. Our notable imports included:

*   `LocalBinaryPatterns`: Responsible for computing LBPs for each input image, thereby quantifying the texture
*   `GridSearchCV`: scikit-learn’s implementation of a grid search for hyperparameter tuning
*   `SVC`: Our Support Vector Machine (SVM) used for classification (SVC)
*   `paths`: Grabs the paths of all images in our input dataset directory
*   `time`: Used to time how long the grid search takes

Next, we have our command line arguments:

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-d", "--dataset", required=True,
	help="path to input dataset")
args = vars(ap.parse\_args())

We only have a single command line argument here, `--dataset`, which will point to our `texture_dataset` residing on disk.

Let’s grab our image paths and initialize our LBP descriptor:

\# grab the image paths in the input dataset directory
imagePaths = list(paths.list\_images(args\["dataset"\]))

# initialize the local binary patterns descriptor along with
# the data and label lists
print("\[INFO\] extracting features...")
desc = LocalBinaryPatterns(24, 8)
data = \[\]
labels = \[\]

**Line 20** grabs the paths to all input images in our `--dataset` directory.

We then initialize our `LocalBinaryPatterns` descriptor, along with two lists:

1.  `data`: Stores the LBPs extracted from each image
2.  `labels`: Contains the class label of the particular image

Let’s populate both `data` and `labels` now:

\# loop over the dataset of images
for imagePath in imagePaths:
	# load the image, convert it to grayscale, and quantify it
	# using LBPs
	image = cv2.imread(imagePath)
	gray = cv2.cvtColor(image, cv2.COLOR\_BGR2GRAY)
	hist = desc.describe(gray)

	# extract the label from the image path, then update the
	# label and data lists
	labels.append(imagePath.split(os.path.sep)\[-2\])
	data.append(hist)

# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
print("\[INFO\] constructing training/testing split...")
(trainX, testX, trainY, testY) = train\_test\_split(data, labels,
	random\_state=22, test\_size=0.25)

On **Line 30,** we loop over our input images.

For each image we:

1.  Load it from disk (**Line 33**)
2.  Convert it to grayscale (**Line 34**)
3.  Compute LBPs for the image (**Line 35**)

We then update our `labels` list with the class label of the particular image along with our `data` list with the computed LBP histogram.

**_Note:_** _Confused on how we determined the class label from the image path? Recall that inside the `texture_dataset` directory, there are three subdirectories, one for each of the three texture classes: brick, marble, and sand. Since the class label of a given image is contained within the file path, all we need to do is extract the subdirectory name, which is exactly what_ **_Line 39_** _does._

Before we can run `GridSearchCV` we first need to define the hyperparameters to search over:

\# construct the set of hyperparameters to tune
parameters = \[
	{"kernel":
		\["linear"\],
		"C": \[0.0001, 0.001, 0.1, 1, 10, 100, 1000\]},
	{"kernel":
		\["poly"\],
		"degree": \[2, 3, 4\],
		"C": \[0.0001, 0.001, 0.1, 1, 10, 100, 1000\]},
	{"kernel":
		\["rbf"\],
		"gamma": \["auto", "scale"\],
		"C": \[0.0001, 0.001, 0.1, 1, 10, 100, 1000\]}
\]

**Line 49** defines a `parameters` list that the grid search will run over. As you can see, we’re testing three different types of SVM kernels: linear, polynomial, and radial basis function (RBF).

Each kernel has its own set of associated hyperparameters to search over as well.

SVMs tend to be _quite sensitive_ to hyperparameter choices; that is _especially true_ for the non-linear kernels. If we want high texture classification accuracy, we _need_ to get these hyperparameter choices correct.

**The values listed above are the ones you’ll typically want to tune for an SVM and given kernel.**

Let’s now run `GridSearchCV` over the hyperparameter space:

\# tune the hyperparameters via a cross-validated GridSearchCV
print("\[INFO\] tuning hyperparameters via gridsearchcv")
grid = GridSearchCV(estimator=SVC(), param\_grid=parameters, n\_jobs=-1)
start = time.time()
grid.fit(trainX, trainY)
end = time.time()

# show GridSearchCV information
print("\[INFO\] GridSearchCV took {:.2f} seconds".format(
	end - start))
print("\[INFO\] GridSearchCV best score: {:.2f}%".format(
	grid.best\_score\_ \* 100))
print("\[INFO\] GridSearchCV best parameters: {}".format(
	grid.best\_params\_))

**Line 65** initializes our `GridSearchCV`, which accepts three parameters:

1.  `estimator`: The model we are tuning (in this case, a Support Vector Machine classifier).
2.  `param_grid`: The hyperparameter space we wish to search (i.e., our `parameters` list).
3.  `n_jobs`: The number of parallel jobs to run. A value of `-1` implies that all processors/cores of your machine will be used, thereby speeding up the `` `GridSearchCV` `` process.

**Line 67** starts the grid search of the hyperparameter space. We wrap the `.fit` call with the `time()` function to measure how long the hyperparameter search space takes.

**Once `` `GridSearchCV` `` is complete, we display three important pieces of information on our terminal:**

1.  How long `` `GridSearchCV` `` took
2.  The best accuracy we obtained during the grid search
3.  The hyperparameters associated with our highest accuracy model

**From there, we do a full evaluation of the best model:**

\# grab the best model and evaluate it
print("\[INFO\] evaluating...")
model = grid.best\_estimator\_
predictions = model.predict(testX)
print(classification\_report(testY, predictions))

**Line 80** grabs the `best_estimator_` from the grid search. **This is the SVM with the highest accuracy.**

**_Note:_** _After a hyperparameter search is complete, the scikit-learn library always populates the `best_estimator_` variable of the `grid` with our highest accuracy model._

**Lines 81** uses the best model found to make predictions on our testing data. We then display a full classification report on **Line 82.**

### **GridSearchCV for computer vision project results**

We are now ready to apply a grid search to tune the hyperparameters to our texture recognition system.

Be sure to access the **_“Downloads”_** section of this tutorial to retrieve the source code and example texture dataset.

From there, you can execute the `train_model.py` script:

$ time python train\_model.py --dataset texture\_dataset
\[INFO\] extracting features...
\[INFO\] constructing training/testing split...
\[INFO\] tuning hyperparameters via gridsearchcv
\[INFO\] GridSearchCV took 1.17 seconds
\[INFO\] GridSearchCV best score: 86.81%
\[INFO\] GridSearchCV best parameters: {'C': 1000, 'degree': 3, 
	'kernel': 'poly'}
\[INFO\] evaluating...
              precision    recall  f1-score   support

       brick       1.00      1.00      1.00        10
      marble       1.00      1.00      1.00         5
        sand       1.00      1.00      1.00         8

    accuracy                           1.00        23
   macro avg       1.00      1.00      1.00        23
weighted avg       1.00      1.00      1.00        23


real	1m39.581s
user	1m45.836s
sys	0m2.896s

As you can see, we’ve obtained **100% accuracy** on our testing set, meaning that our SVM was capable of recognizing the texture inside every one of our images.

Furthermore, running the tuning script took only 1m39s.

A grid search worked well here, but as I mentioned in [last week’s tutorial](https://pyimagesearch.com/2021/05/17/introduction-to-hyperparameter-tuning-with-scikit-learn-and-python/), a random search tends to work just as well and requires less time to run — the more hyperparameters in your search space, the longer ``` `` `GridSearchCV` `` ``` takes (growing exponentially).

To illustrate this point, next week, I’ll show you how to use a random search to tune the hyperparameters in a deep learning model.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial, you learned how to use a ``` `` `GridSearchCV` `` ``` to tune hyperparameters to a machine learning model automatically. To implement the grid search, we used the scikit-learn library and the `GridSearchCV` class.

Our goal was to train a computer vision model that can automatically recognize the texture of an object in an image (brick, marble, or sand).

The training pipeline itself included:

1.  Looping over all images in our dataset
2.  Quantifying the texture of each image using the [Local Binary Patterns descriptor](https://pyimagesearch.com/2015/12/07/local-binary-patterns-with-python-opencv/) (a popular image descriptor often used for quantifying texture)
3.  Using a grid search to explore hyperparameters to our Support Vector Machine

After tuning our SVM hyperparameters, we obtained 100% classification accuracy on our texture recognition dataset.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!