Title: How to tune hyperparameters with Python and scikit-learn - PyImageSearch

URL Source: https://pyimagesearch.com/2016/08/15/how-to-tune-hyperparameters-with-python-and-scikit-learn/

Published Time: 2016-08-15T14:00:00+00:00

Markdown Content:
![Image 1: tune_hyperparams_header](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20129'%3E%3C/svg%3E)

In last week’s post, I introduced the [k-NN machine learning algorithm](https://pyimagesearch.com/2016/08/08/k-nn-classifier-for-image-classification/) which we then applied to the task of image classification.

Using the k-NN algorithm, we obtained **57.58% classification accuracy** on the Kaggle Dogs vs. Cats dataset challenge:

![Image 2: Figure 1: Classifying an image as whether or contains a dog or a cat.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20485'%3E%3C/svg%3E)

**Figure 1:** Classifying an image as whether it contains a dog or a cat.

**The question is: _“Can we do better?”_**

Of course we can! Obtaining higher accuracy for nearly any machine learning algorithm boils down to tweaking various knobs and levels.

In the case of k-NN, we can tune _k_, the number of nearest neighbors. We can also tune our distance metric/similarity function as well.

Of course, hyperparameter tuning has implications _outside_ of the k-NN algorithm as well. In the context of Deep Learning and Convolutional Neural Networks, we can easily have _hundreds_ of various hyperparameters to tune and play with (although in practice we try to limit the number of variables to tune to a small handful), each affecting our overall classification to some (potentially unknown) degree.

Because of this, it’s important to understand the concept of _hyperparameter tuning_ and how your choice in hyperparameters can _dramatically_ impact your classification accuracy.

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the remainder of today’s tutorial, I’ll be demonstrating how to tune k-NN hyperparameters for the [Dogs vs. Cats dataset](https://www.kaggle.com/c/dogs-vs-cats/data). We’ll start with a discussion on _what hyperparameters are_, followed by viewing a concrete example on _tuning k-NN hyperparameters._

We’ll then explore how to tune k-NN hyperparameters using two search methods: _Grid Search_ and _Randomized Search._

As our results will demonstrate, we can improve our classification accuracy from **57.58% to over 64%!**

### What are hyperparameters?

Hyperparameters are simply the knobs and levels you pull and turn when building a machine learning classifier. The process of tuning hyperparameters is more formally called _[hyperparameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization)_.

So what’s the difference between a normal “model parameter” and a “hyperparameter”?

Well, a standard “model parameter” is normally an internal variable that is optimized in some fashion. In the context of Linear Regression, Logistic Regression, and Support Vector Machines, we would think of parameters as the weight vector coefficients found by the learning algorithm.

On the other hand, “hyperparameters” are normally set by a human designer or tuned via algorithmic approaches. Examples of hyperparameters include the number of neighbors _k_ in the k-Nearest Neighbor algorithm, the learning rate _alpha_ of a Neural Network, or the number of filters learned in a given convolutional layer in a CNN.

In general, model parameters are optimized according to some _loss function_, while hyperparameters are instead _searched for_ by exploring various settings to see which values provided the highest level of accuracy.

Because of this, it tends to be easier to tune model parameters (since we’re optimizing some objective function based on our training data) whereas hyperparameters can require a nearly blind search to find optimal ones.

### k-NN hyperparameters

As a concrete example of tuning hyperparameters, let’s consider the [k-Nearest Neighbor classification algorithm](https://pyimagesearch.com/2016/08/08/k-nn-classifier-for-image-classification/). For your standard k-NN implementation, there are _two primary hyperparameters_ that you’ll want to tune:

1.  The number of neighbors _k_.
2.  The distance metric/similarity function.

Both of these values can dramatically affect the accuracy of your k-NN classifier. To demonstrate this in the context of image classification, let’s apply hyperparameter tuning to our Kaggle Dogs vs. Cats dataset from last week.

Open up a new file, name it `knn_tune.py` , and insert the following code:

\# import the necessary packages
from sklearn.neighbors import KNeighborsClassifier
from sklearn.grid\_search import RandomizedSearchCV
from sklearn.grid\_search import GridSearchCV
from sklearn.cross\_validation import train\_test\_split
from imutils import paths
import numpy as np
import argparse
import imutils
import time
import cv2
import os

**Lines 2-12** start by importing our required Python packages. We’ll be making heavy use of the [scikit-learn library](http://scikit-learn.org/stable/), so if you do not have it installed, [make sure you follow these instructions](http://scikit-learn.org/stable/install.html).

We’ll also be using my personal [imutils library](https://github.com/jrosebr1/imutils), so make sure you have it installed as well:

$ pip install imutils

Next, we’ll define our `extract_color_histogram` function:

def extract\_color\_histogram(image, bins=(8, 8, 8)):
	# extract a 3D color histogram from the HSV color space using
	# the supplied number of \`bins\` per channel
	hsv = cv2.cvtColor(image, cv2.COLOR\_BGR2HSV)
	hist = cv2.calcHist(\[hsv\], \[0, 1, 2\], None, bins,
		\[0, 180, 0, 256, 0, 256\])

	# handle normalizing the histogram if we are using OpenCV 2.4.X
	if imutils.is\_cv2():
		hist = cv2.normalize(hist)

	# otherwise, perform "in place" normalization in OpenCV 3 (I
	# personally hate the way this is done
	else:
		cv2.normalize(hist, hist)

	# return the flattened histogram as the feature vector
	return hist.flatten()

This function accepts an input `image` along with a number of `bins` for each channel of the image.

We convert the image to the HSV color space and compute a 3D color histogram to characterize the color distribution of the image (**Lines 17-19**).

This histogram is then flattened into a single _8 x 8 x 8 = 512-d_ feature vector that is returned to the calling function.

For a more detailed review of this method, [please refer to last week’s blog post](https://pyimagesearch.com/2016/08/08/k-nn-classifier-for-image-classification/).

\# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-d", "--dataset", required=True,
	help="path to input dataset")
ap.add\_argument("-j", "--jobs", type=int, default=-1,
	help="# of jobs for k-NN distance (-1 uses all available cores)")
args = vars(ap.parse\_args())

# grab the list of images that we'll be describing
print("\[INFO\] describing images...")
imagePaths = list(paths.list\_images(args\["dataset"\]))

# initialize the data matrix and labels list
data = \[\]
labels = \[\]

**Lines 34-39** handle parsing our command line arguments. We only need two switches here:

*   `--dataset` : The path to our input Dogs vs. Cats dataset from the Kaggle challenge.
*   `--jobs` : The number of processors/cores to utilize when computing the nearest neighbors for a particular data point. Setting this value to `-1` indicates _all_ available processors/cores should be used. Again, for a more detailed review of these arguments, [please refer to last week’s tutorial.](https://pyimagesearch.com/2016/08/08/k-nn-classifier-for-image-classification/)

**Line 43** grabs the paths to our 25,000 input images while **Lines 46 and 47** initializes the `data` list (where we’ll store the color histogram extracted from each image) and `labels` list (either “dog” or “cat” for each input image), respectively.

Next, we can loop over our `imagePaths` and describe them:

\# loop over the input images
for (i, imagePath) in enumerate(imagePaths):
	# load the image and extract the class label (assuming that our
	# path as the format: /path/to/dataset/{class}.{image\_num}.jpg
	image = cv2.imread(imagePath)
	label = imagePath.split(os.path.sep)\[-1\].split(".")\[0\]

	# extract a color histogram from the image, then update the
	# data matrix and labels list
	hist = extract\_color\_histogram(image)
	data.append(hist)
	labels.append(label)

	# show an update every 1,000 images
	if i > 0 and i % 1000 == 0:
		print("\[INFO\] processed {}/{}".format(i, len(imagePaths)))

**Line 50** starts looping over each of the `imagePaths` . For each `imagePath` , we load it from disk and extract the `label` (**Lines 53 and 54**).

Now that we have our `image` , we compute a color histogram (**Line 58**), followed by updating the `data` and `labels` lists (**Lines 59 and 60**).

Finally, **Lines 63 and 64** display the feature extraction progress to our screen.

In order to train and evaluate our k-NN classifier, we’ll need to partition our `data` into two splits: a _training split_ and a _testing split_:

\# partition the data into training and testing splits, using 75%
# of the data for training and the remaining 25% for testing
print("\[INFO\] constructing training/testing split...")
(trainData, testData, trainLabels, testLabels) = train\_test\_split(
	data, labels, test\_size=0.25, random\_state=42)

Here we’ll be using 75% of our data for training and the remaining 25% for evaluation.

Finally, let’s define the set of hyperparameters we are going to optimize over:

\# construct the set of hyperparameters to tune
params = {"n\_neighbors": np.arange(1, 31, 2),
	"metric": \["euclidean", "cityblock"\]}

The above code block defines a `params` dictionary which contains two keys:

*   `n_neighbors` : The number of nearest neighbors _k_ in the k-NN algorithm. Here we’ll search over the _odd integers_ in the range _\[0, 29\]_ (keep in mind that the `np.arange` function is _exclusive_).
*   `metric` : This is the distance function/similarity metric for k-NN. Normally this defaults to the Euclidean distance, but we could also use any function that returns a single floating point value representing how “similar” two images are. In this case, we’ll search over both the [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) and [Manhattan/City block distance](https://en.wikipedia.org/wiki/Taxicab_geometry).

Now that we have defined the hyperparameters we want to search over, we need a method that actually _applies_ the search. Luckily, the scikit-learn library already has two methods that can perform hyperparameter search for us: **Grid Search** and **Randomized Search**.

As we’ll find out, it’s normally preferable to used Randomized Search over Grid Search in nearly all circumstances.

### Grid Search hyperparameters

The Grid Search tuning algorithm will methodically (and exhaustively) train and evaluate a machine learning classifier for **_each and every combination_** of hyperparameter values.

In this case, given 16 unique values of _k_ and 2 unique values for our _distance metric_, a Grid Search will apply _**30 different experiments to determine the optimal value.**_

You can see how a Grid Search is performed in the following code segment:

\# tune the hyperparameters via a cross-validated grid search
print("\[INFO\] tuning hyperparameters via grid search")
model = KNeighborsClassifier(n\_jobs=args\["jobs"\])
grid = GridSearchCV(model, params)
start = time.time()
grid.fit(trainData, trainLabels)

# evaluate the best grid searched model on the testing data
print("\[INFO\] grid search took {:.2f} seconds".format(
	time.time() - start))
acc = grid.score(testData, testLabels)
print("\[INFO\] grid search accuracy: {:.2f}%".format(acc \* 100))
print("\[INFO\] grid search best parameters: {}".format(
	grid.best\_params\_))

The primary benefit of the Grid Search algorithm is also it’s major drawback: as an _exhaustive search_ your number of possible parameter values _explodes_ as both the number of _hyperparameters and hyperparameter values_ increases.

Sure, you get to evaluate each and every combination of hyperparameter — **but you pay a cost** — it’s a _very_ time consuming cost. And in most cases, _it’s hardly worth it._

As I explain in the _“Use Randomized Search for hyperparameter tuning (in most situations)”_ section below, there are _rarely_ just _one_ set of hyperparameters that obtain the highest accuracy.

Instead, there are “hot zones” of hyperparameters that all obtain near identical accuracy. The goal is to explore as many of these “zones” of hyperparameters a quickly as possible and locate one of these “hot zones”. It turns out that a random search is a great way to do this.

### Randomized Search hyperparameters

The Random Search approach to hyperparameter tuning will _sample_ hyperparameters from our `params` dictionary via a **random, uniform distribution.** Given a set of randomly sampled parameters, a model is then trained and evaluated.

We perform this set of random hyperparameter sampling and model construction/evaluation for a _preset number of times._ You set the number of evaluations to be **as long as you’re willing to wait**. If you’re impatient and in a hurry, make this value low. And if you have the time to spend on a longer experiment, increase the number of iterations.

In either case, the goal of a Randomized Search is to explore a large set of possible hyperparameter spaces quickly — and the best way to accomplish this is via simple random sampling. **And in practice, it works quite well!**

You can find the code to perform a Randomized Search of hyperparameters for the k-NN algorithm below:

\# tune the hyperparameters via a randomized search
grid = RandomizedSearchCV(model, params)
start = time.time()
grid.fit(trainData, trainLabels)

# evaluate the best randomized searched model on the testing
# data
print("\[INFO\] randomized search took {:.2f} seconds".format(
	time.time() - start))
acc = grid.score(testData, testLabels)
print("\[INFO\] grid search accuracy: {:.2f}%".format(acc \* 100))
print("\[INFO\] randomized search best parameters: {}".format(
	grid.best\_params\_))

### Hyperparameter tuning with Python and scikit-learn results

To tune the hyperparameters of our k-NN algorithm, make sure you:

1.  Download the source code to this tutorial using the **_“Downloads”_** form at the bottom of this post.
2.  Head over to the [Kaggle Dogs vs. Cats competition page](https://www.kaggle.com/c/dogs-vs-cats/data) and download the dataset.

From there, you can execute the following command to tune the hyperparameters:

$ python knn\_tune.py --dataset kaggle\_dogs\_vs\_cats

You’ll probably want to go for a nice walk and stretch your legs will the `knn_tune.py`  script executes.

On my machine, it took **19m 26s** to complete, with over 86% of this time spent Grid Searching:

![Image 4: Figure 2: Applying a Grid Search and Randomized to tune machine learning hyperparameters using Python and scikit-learn.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20457'%3E%3C/svg%3E)

**Figure 2:** Applying a Grid Search and Randomized to tune machine learning hyperparameters using Python and scikit-learn.

As you can see from the output screenshot, the Grid Search method found that _k=25_ and _metric=’cityblock’_ obtained the highest accuracy of 64.03%. **However, this Grid Search took 13 minutes.**

On the other hand, the Randomized Search obtained an identical accuracy of 64.03% — **and it completed in _under 5 minutes._**

Both of these hyperparameter tuning methods improved our classification accuracy (64.03% accuracy, up from 57.58% [from last week’s post](https://pyimagesearch.com/2016/08/08/k-nn-classifier-for-image-classification/)) — but the Randomized Search was _much_ more efficient.

### Use Randomized Search for hyperparameter tuning (in most situations)

Unless your search space is small and can easily be enumerated, a Randomized Search will tend to be more efficient and yield _better results faster._

As our experiments demonstrated, Randomized Search was able to obtain **64.03% accuracy in < 5 minutes** while an exhaustive Grid Search took a much longer **13 minutes to obtain an identical 64.03% accuracy** — that’s a 202% increase in evaluation time for identical accuracy!

In general, there isn’t just _one_ set of hyperparameters that obtains optimal results — instead, there are usually a _set_ of them that exist towards the bottom of a concave bowl (i.e., the optimization surface).

As long as you hit just _one_ of these parameters towards the bottom of the bowl, you’ll still obtain the same accuracy as if you enumerated _all possibilities_ along the bowl. Furthermore, you’ll be able to explore various regions of this bowl _faster_ by applying a Randomized Search.

Overall, this will lead to faster, more efficient hyperparameter tunings in most situations.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In today’s blog post, I demonstrated how to tune hyperparameters to machine learning algorithms using the Python programming language and the scikit-learn library.

First, I defined, the difference between standard “model parameters” and the “hyperparameters” that need to be tuned.

From there, we applied two methods to tune hyperparameters:

1.  An exhaustive Grid Search
2.  A Randomized Search

Both of these hyperparameter tuning routines were then applied to the k-NN algorithm and the Kaggle Dogs vs. Cats dataset.

Each respective tuning algorithm obtained identical accuracy — _but the Randomized Search was able to obtain this increase of accuracy in a fraction of the time!_

**In general, I _highly encourage_ you to use Randomized Search when tuning hyperparameters.** You’ll often find that there is rarely just _one_ set of hyperparameters that obtains optimal accuracy. Instead, there are “hot zones” of hyperparameters that will obtain near-identical accuracy — the goal is to explore as many zones and try to land on one of these zones as fast as possible.

Given no _a priori_ knowledge of good hyperparameter choices, a Randomized Search to hyperparameter tuning is the most optimal way to find reasonable hyperparameter values in a short amount of time as it allows you to explore many areas of the optimization surface.

Anyway, I hope you enjoyed this blog post! I’ll be back next week to discuss the basics of linear classification (and the role it plays in Neural Networks and image classification).

**But before you go, _be sure to signup for the PyImageSearch Newsletter using the form below to be notified when future blog posts are published!_**

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!