Title: Introduction to hyperparameter tuning with scikit-learn and Python - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/17/introduction-to-hyperparameter-tuning-with-scikit-learn-and-python/

Published Time: 2021-05-17T14:00:00+00:00

Markdown Content:
In this tutorial, you will learn how to tune machine learning model hyperparameters with scikit-learn and Python.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20490'%3E%3C/svg%3E)

This tutorial is part one in a four-part series on hyperparameter tuning:

1.  _Introduction to hyperparameter tuning with scikit-learn and Python_ (today’s post)
2.  _Grid search hyperparameter tuning with scikit-learn ( GridSearchCV )_ (next week’s post)
3.  _Hyperparameter tuning for Deep Learning with scikit-learn, Keras, and TensorFlow_ (tutorial two weeks from now)
4.  _Easy Hyperparameter Tuning with Keras Tuner and TensorFlow_ (final post in the series)

**Tuning your hyperparameters is** **_absolutely critical_** **in obtaining a high-accuracy model.** Many machine learning models have various knobs, dials, and parameters that you can set.

The difference between a very low-accuracy model versus a high-accuracy one is sometimes as simple as tuning the right dial. **This tutorial will show you how to tune the dials on your machine learning model to boost your accuracy.**

Specifically, we’ll be covering the basing of hyperparameter tuning by:

1.  Obtaining a **baseline** with _no hyperparameter tuning_ where we have a benchmark to improve
2.  Exhaustively **grid searching** over a set of hyperparameters
3.  Utilizing a **random search** to sample from a hyperparameter space

We’ll implement each method using Python and scikit-learn, train our model, and evaluate the results.

By the end of this tutorial, you’ll have a strong understanding of how to practically use hyperparameter tuning in your own projects to boost model accuracy.

**To learn how to tune hyperparameters with scikit-learn and Python,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In this tutorial, you will learn how to tune model hyperparameters using scikit-learn and Python.

We’ll start the tutorial by discussing what hyperparameter tuning is and why it’s so important.

From there, we’ll configure your development environment and review the project directory structure.

We’ll then have three Python scripts to implement:

1.  One that trains a model with _no_ hyperparameter tuning (so we can obtain a baseline)
2.  One that utilizes an algorithm called “grid search” to exhaustively examine _all combinations_ of hyperparameters — this method is guaranteed to do a full sweep of hyperparameter values, but also _very slow_
3.  And a final method that uses a “random search” by sampling various hyperparameters from a distribution (not guaranteed to cover all hyperparameter values but in practice is typically just as accurate as grid search and runs _far faster_)

Finally, we’ll wrap up our tutorial with a discussion of the results.

### **What is hyperparameter tuning, and why is it important?**

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20616'%3E%3C/svg%3E)

**Figure 1:** Tuning the hyperparameters to your model is a requirement to obtain optimal accuracy.

Have you ever tried using an old-school AM/FM radio before? You know, the analog radios that have knobs and dials you can move to select a station, adjust the equalizer, etc., like in **Figure 1**?

These radios may be a relic of the past, but if you’ve ever used one before, you know how important it is to get that station selector “just right.” **If you let the station selector sit between two frequencies, you’ll get audio from two different stations bleeding into each other.** The noise is cacophonous, and it’s near impossible to understand.

Similarly, machine learning models have _all sorts_ of knobs and dials you can tune:

*   Neural Networks have learning rate and regularization strength
*   Convolutional Neural Networks have several layers, number of filters per convolutional layer, number of nodes in the fully connected layers, etc.
*   Decision trees have the node split criteria (Gini index, information gain, etc.)
*   Random Forests have the total number of trees in the forest, along with feature space sampling percentages
*   Support Vector Machines (SVMs) have the type of kernel (linear, polynomial, radial basis function (RBF), etc.) along with any parameters you need to tune for the particular kernel

**SVMs are** **_notorious_** **for requiring significant hyperparameter tuning,** **_especially_** **if you are using a non-linear kernel.** Not only do you need to select the correct type of kernel for your data, but then you also need to tune any knobs and dials associated with the kernel — _one wrong choice, and your accuracy can plummet._

### **Scikit-learn: hyperparameter tuning with grid search and random search**

The two hyperparameter methods you’ll use most frequently with scikit-learn are a **grid search** and a **random search.** The general idea behind both of these algorithms is that you:

1.  Define a set of hyperparameters you want to tune
2.  Give these hyperparameters to the grid search or random search
3.  These algorithms then _automatically_ examine the hyperparameter search space and attempt to find the optimal values that maximize accuracy

That said, grid search and random search are inherently different techniques to hyperparameter tuning. **Figure 2** visualizes these two hyperparameter search algorithms:

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20342'%3E%3C/svg%3E)

**Figure 2:** A visualization of performing a grid search versus a random search to tune hyperparameters ([image source](https://medium.com/@jackstalfort/hyperparameter-tuning-using-grid-search-and-random-search-f8750a464b35)).

In **Figure 2**, we have a 2D grid with values of the first hyperparameter plotted along the _x_\-axis and values of the second hyperparameter on the _y-_axis. **The white highlighted oval is where the optimal values for _both_ these hyperparameters lie. Our goal is to locate this region using our hyperparameter tuning algorithms.**

**Figure 2** (_left)_ visualizes a grid search:

1.  We start by defining a set of all hyperparameters and the associated values we want to explore
2.  The grid search then examines **_all combinations_** of these hyperparameters
3.  For each possible combination of hyperparameters, we train a model on them
4.  The hyperparameters associated with the highest accuracy are then returned

A grid search is guaranteed to examine all possible combinations of hyperparameters. **The problem is that the more hyperparameters you have, the more the number of combinations grows** **_exponentially!_**

And since there are so many combinations to examine, a grid search tends to run _very slowly._

To help speed the process, we can employ a **random search** (**Figure 2**, _right_).

With a random search, we:

1.  Define the hyperparameters we want to search over
2.  Set a lower and upper bound on the values (if it’s a continuous variable) or the possible values the hyperparameter can take on (if it’s a categorical variable) for each hyperparameter
3.  A random search then _randomly samples_ from these distributions a total of _N_ times, training a model on each set of hyperparameters
4.  The hyperparameters associated with the highest accuracy model are then returned

While a random search, by definition, is not an exhaustive search like the grid search, the benefit of a random search is that it’s **_far faster_** and typically obtains just as high accuracy as a grid search.

Why is that, though?

Shouldn’t a grid search obtain the highest accuracy since it’s so exhaustive?

Not necessarily.

**When tuning hyperparameters, there isn’t just** **_one_** **“golden set of values” that will give you the highest accuracy. Instead, it’s a** **_distribution_** **— there are** **_ranges_** **for each hyperparameter that will obtain the best accuracy**. If you land within that range/distribution, you’ll still enjoy the same high accuracy without the requirement of exhaustively tuning your hyperparameters with a grid search.

### **Configuring your development environment**

To follow this guide, you need to have the scikit-learn machine library and Pandas data science library installed on your system.

Luckily, both of these packages are pip-installable:

$ pip install scikit-learn
$ pip install pandas

If you’re having a problem configuring your development environment, you should refer to the next section.

### **Having problems configuring your development environment?**

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 3:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux systems?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

### **Our example dataset**

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20467'%3E%3C/svg%3E)

**Figure 4:** We’ll use the Abalone dataset from the UCI Machine Learning Repository to tune our hyperparameters ([image source](https://culturedabalone.com/product/20-pack/)).

The dataset we’ll be using today is the [Abalone from the UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/abalone).

Abalones are marine snails often studied by marine biologists. Specifically, marine biologists are interested in the _age_ of the snail.

The _problem_ is that determining the age of an abalone is a time consuming, tedious task, requiring the biologist to:

1.  Cut the abalone shell through the cone
2.  Stain it
3.  Count the number of rings on the stained shell using a microscope (like measuring rings on a tree)

Yes, it’s just as boring as it sounds.

However, using _more easily gathered values_, it’s possible to _predict_ the marine snail’s age.

The Abalone dataset includes seven values:

1.  Length of the shell
2.  Diameter of the shell
3.  Height of the shell
4.  Weight (entire weight of the abalone)
5.  Shucked weight (weight of the abalone meat)
6.  Viscera weight
7.  Shell weight

Given these seven values, our goal is to train a machine learning model to predict the abalone’s age. **As we’ll see, tuning hyperparameters to our machine learning model can lead to a boost in accuracy.**

### **Project structure**

Let’s start by reviewing our project directory structure.

Be sure to access the **_“Downloads”_** section of this tutorial to retrieve the source code.

You’ll then be presented with the following files:

$ tree . --dirsfirst
.
├── pyimagesearch
│   └── config.py
├── abalone\_train.csv
├── train\_svr.py
├── train\_svr\_grid.py
└── train\_svr\_random.py

1 directory, 5 files

The `abalone_train.csv` file contains the [Abalone dataset from the UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/abalone) (there is no need to download the CSV file from UCI as I have included the CSV file in the downloads associated with the tutorial).

The Abalone dataset contains 3,320 rows with eight columns per row (seven for the features, including shell length, diameter, etc.) and a final column for the age (which is the target value we seek to predict).

To evaluate the impact hyperparameter tuning has, we’ll be implementing three Python scripts:

1.  `train_svr.py`: Establishes a baseline on the abalone dataset by training a Support Vector Regression (SVR) with _no hyperparameter tuning._
2.  `train_svr_grid.py`: Utilizes a grid search for hyperparameter tuning.
3.  `train_svr_random.py`: Performs a random hyperparameter search over the distribution of hyperparameter values.

The `config.py` file inside the `pyimagesearch` module implements our important configuration variables used across the three driver scripts, including the path to the input CSV dataset and the column names.

### **Creating our configuration file**

Before we can implement any training scripts, let’s first review our configuration file.

Open the `config.py` file in the `pyimagesearch` module, and you’ll find the following code:

\# specify the path of our dataset
CSV\_PATH = "abalone\_train.csv"

# specify the column names of our dataframe
COLS = \["Length", "Diameter", "Height", "Whole weight",
	"Shucked weight", "Viscera weight", "Shell weight", "Age"\]

On **Line 2,** we initialize the `CSV_PATH` to point to our Abalone CSV file residing on disk.

**Lines 5 and 6** then define the names of each of the eight columns in the CSV file. We’ll need the column names in our training script to load it from disk and parse it.

That’s it for our configuration file! Let’s move on to implementing our training scripts.

### **Implementing a basic training script**

Our first training script, `train_svr.py`, will establish a **baseline accuracy** by performing _no hyperparameter tuning._ Once we have this baseline, we’ll attempt to improve on it by applying hyperparameter tuning algorithms.

But for now, let’s establish our baseline.

Open the `train_svr.py` file in your project directory, and let’s get to work:

\# import the necessary packages
from pyimagesearch import config
from sklearn.preprocessing import StandardScaler
from sklearn.model\_selection import train\_test\_split
from sklearn.svm import SVR
import pandas as pd

**Lines 2-6** import our required Python packages, including:

*   `config`: The configuration file we implemented in the previous section
*   `StandardScaler`: A data preprocessing technique that performs standard scaling (also known as _z_\-scores in the statistics world) that scales each data observation by subtracting the mean of the particular column values divided by the standard deviation
*   `train_test_split`: Constructs a training and testing split
*   `SVR`: scikit-learn’s implementation of a Support Vector Machine used for regression
*   `pandas`: Used to load our CSV file from disk and parse the data

Speaking of which, let’s load our Abalone CSV file from disk now:

\# load the dataset, separate the features and labels, and perform a
# training and testing split using 85% of the data for training and
# 15% for evaluation
print("\[INFO\] loading data...")
dataset = pd.read\_csv(config.CSV\_PATH, names=config.COLS)
dataX = dataset\[dataset.columns\[:-1\]\]
dataY = dataset\[dataset.columns\[-1\]\]
(trainX, testX, trainY, testY) = train\_test\_split(dataX,
	dataY, random\_state=3, test\_size=0.15)

**Line 12** reads our CSV file from disk. Notice how we pass in the `CSV_PATH` to the file on disk along with the `names` of the columns.

From there, we need to extract our feature vectors (i.e., length of the shell, diameter, height, etc.) and the target value we wish to predict (age) — **Lines 13 and 14** do that for us via simple array slicing.

Once our data has been loaded and parsed, we construct a training and testing split on **Lines 15 and 16** using 85% of the data for training and 15% for testing.

With our data loaded we now need to preprocess it:

\# standardize the feature values by computing the mean, subtracting
# the mean from the data points, and then dividing by the standard
# deviation
scaler = StandardScaler()
trainX = scaler.fit\_transform(trainX)
testX = scaler.transform(testX)

**Line 21** instantiates an instance of our `StandardScaler`. We then compute the mean and standard deviation of all column values in our training data and then scale them (**Line 22**).

**Line 23** scales our testing data using the mean and standard deviation computed over the training set.

All that’s left now is to train the SVR:

\# train the model with \*no\* hyperparameter tuning
print("\[INFO\] training our support vector regression model")
model = SVR()
model.fit(trainX, trainY)

# evaluate our model using R^2-score (1.0 is the best value)
print("\[INFO\] evaluating...")
print("R2: {:.2f}".format(model.score(testX, testY)))

**Line 27** instantiates our Support Vector Machine for regression. Remember, we are using regression here because we are trying to predict a real-valued output, **the** **_age_** **of the abalone.**

**Line 28** then trains the model using our `trainX` (feature vectors) and `trainY` (target ages to predict).

Once our model is trained, we then evaluate it on **Line 32** using the coefficient of determination.

The coefficient of determination is typically used when analyzing the output of a regression model. Essentially, it measures the amount of “explainable variance” between the dependent variable and the independent variable.

The coefficient will have a value in the range _\[0, 1\],_ where _0_ implies that we _cannot_ predict the target output correctly, while a value of _1_ implies we can perfectly predict the output with no error.

For more information on the coefficient of determination, be sure to refer to [this page](https://stattrek.com/statistics/dictionary.aspx?definition=coefficient_of_determination).

### **Gathering our baseline results (****_no_** **hyperparameter tuning)**

Before we can tune hyperparameters to our SVM, we first need to obtain a baseline with _no_ hyperparameter tuning. Doing so will give us a baseline/benchmark where we can improve.

Start by accessing the **_“Downloads”_** section of this tutorial to retrieve the source code and example dataset.

You can then execute the `train_svr.py` script:

$ time python train\_svr.py
\[INFO\] loading data...
\[INFO\] training our support vector regression model
\[INFO\] evaluating...
R2: 0.55

real	0m1.386s
user	0m1.928s
sys	0m1.040s

Here we obtain a coefficient of determination value of `0.55`, meaning that 55% of the variance in _Y_ (the age of the marine snails) is predictable based on _X_ (our seven feature values).

**Our goal is to use hyperparameter tuning to beat this value.**

### **Tuning hyperparameters with a grid search**

Now that we’ve established a baseline score, let’s see if we can beat it using hyperparameter tuning with scikit-learn.

We’ll start by implementing a grid search. Later in this guide, you’ll also learn how to use a random search for hyperparameter tuning as well.

Open the `train_svr_grid.py` file in your project directory structure, and we’ll implement a grid search using scikit-learn:

\# import the necessary packages
from pyimagesearch import config
from sklearn.model\_selection import RepeatedKFold
from sklearn.model\_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR
from sklearn.model\_selection import train\_test\_split
import pandas as pd

**Lines 2-8** import our required Python packages. These imports are identical to our previous `train_svr.py` script but with two additions:

1.  `GridSearchCV`: scikit-learn’s implementation of the grid search hyperparameter tuning algorithm
2.  `RepeatedKFold`: Performs _k_\-fold cross-validation a total of _N_ times using different randomization at each iteration

Performing _k-_fold cross-validation allows us to _“improve the estimated performance of a machine learning model”_ ([source](https://machinelearningmastery.com/repeated-k-fold-cross-validation-with-python/)) and is typically utilized when performing hyperparameter tuning.

The _last_ thing you want when tuning hyperparameters is to run a long experiment on a randomized set of data, obtain high accuracy, and then find the high accuracy was due to an anomaly in the randomization of the data itself. Utilizing cross-validation helps prevent that from happening.

We can now load our CSV file from disk and preprocess it:

\# load the dataset, separate the features and labels, and perform a
# training and testing split using 85% of the data for training and
# 15% for evaluation
print("\[INFO\] loading data...")
dataset = pd.read\_csv(config.CSV\_PATH, names=config.COLS)
dataX = dataset\[dataset.columns\[:-1\]\]
dataY = dataset\[dataset.columns\[-1\]\]
(trainX, testX, trainY, testY) = train\_test\_split(dataX,
	dataY, random\_state=3, test\_size=0.15)

# standardize the feature values by computing the mean, subtracting
# the mean from the data points, and then dividing by the standard
# deviation
scaler = StandardScaler()
trainX = scaler.fit\_transform(trainX)
testX = scaler.transform(testX)

This code was covered in the _“Implementing a basic training script_” section above, so refer there if you need additional details on what the code is doing.

Next, we can initialize our regression model and hyperparameter search space:

\# initialize model and define the space of the hyperparameters to
# perform the grid-search over
model = SVR()
kernel = \["linear", "rbf", "sigmoid", "poly"\]
tolerance = \[1e-3, 1e-4, 1e-5, 1e-6\]
C = \[1, 1.5, 2, 2.5, 3\]
grid = dict(kernel=kernel, tol=tolerance, C=C)

**Line 29** initializes our Support Vector Machine regression (SVR) model. An SVR has several hyperparameters to tune, including:

1.  `kernel`: The type of kernel used when projecting the data into a higher-dimensional space where it ideally becomes linearly separable
2.  `tolerance`: The tolerance for stopping criterion
3.  `C`: The “strictness” of the SVR (i.e., to what degree the SVR is allowed to make mistakes when fitting the data)

**Lines 30-32** define values for each of these hyperparameters, while **Line 33** creates a dictionary of the hyperparameters.

**Our goal will be to search these hyperparameters and find the optimal values for `kernel`, `tolerance`, and `C`.**

Speaking of which, let’s perform our grid search now:

\# initialize a cross-validation fold and perform a grid-search to
# tune the hyperparameters
print("\[INFO\] grid searching over the hyperparameters...")
cvFold = RepeatedKFold(n\_splits=10, n\_repeats=3, random\_state=1)
gridSearch = GridSearchCV(estimator=model, param\_grid=grid, n\_jobs=-1,
	cv=cvFold, scoring="neg\_mean\_squared\_error")
searchResults = gridSearch.fit(trainX, trainY)

# extract the best model and evaluate it
print("\[INFO\] evaluating...")
bestModel = searchResults.best\_estimator\_
print("R2: {:.2f}".format(bestModel.score(testX, testY)))

**Line 38** creates our cross-validation fold, indicating that we will generate `10` folds and then repeat the entire process a total of `3` times (typically, you’ll see fold and repeat values in the range _\[3, 10\]_).

**Lines 39 and 40** instantiate our `GridSearchCV` object. Let’s break down each of the arguments:

*   `estimator`: The model we’re attempting to optimize (i.e., our SVR that will predict the age of the abalone)
*   `param_grid`: The hyperparameter search space
*   `n_jobs`: The number of cores on your processor(s) that will be used to run parallel jobs. A value of `-1` implies that _all_ cores/processors will be used, thereby speeding the grid search process.
*   `scoring`: The loss function we’re attempting to optimize; in this case, we are trying to drive down our mean squared error (MSE), implying that the _lower_ the MSE, the _better_ our model is at predicting the age of the abalone

**Line 41** kicks off the grid search.

After the grid search runs, we obtain the `bestModel` found during the search (**Line 45**) and then compute the determination coefficient, which will tell us how good of a job our model did (**Line 46**).

### **Grid search hyperparameter tuning results**

Let’s put the grid search hyperparameter tuning method to the test.

Be sure to access the **_“Downloads”_** section of this tutorial to retrieve the source code and example dataset.

From there, you can execute the following command:

$ time python train\_svr\_grid.py
\[INFO\] loading data...
\[INFO\] grid searching over the hyperparameters...
\[INFO\] evaluating...
R2: 0.56

real	4m34.825s
user	35m47.816s
sys	0m37.268s

Previously, no hyperparameter tuning yielded a coefficient of determination value of `0.55`.

**Using a grid search, we have improved that value to `0.56`, implying that our tuned model is doing a better job predicting snail age.**

However, obtaining this accuracy comes at the cost of speed:

*   It only took 1.3 seconds to train our original model with no hyperparameter tuning
*   An exhaustive grid search took 4m34s
*   **That’s an increase of 18,166%**

**And what’s worse is that the** **_more_** **hyperparameters you want to tune, the number of possible value combinations explodes** **_exponentially_** — for real-world datasets and problems, that’s just not feasible.

The solution is to utilize a random search to tune your hyperparameters.

### **Tuning hyperparameters with a randomized search**

Our final experiment will explore using a randomized hyperparameter search.

Open the `train_svr_random.py` file, and we’ll get started:

\# import the necessary packages
from pyimagesearch import config
from sklearn.model\_selection import RandomizedSearchCV
from sklearn.model\_selection import RepeatedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR
from sklearn.model\_selection import train\_test\_split
from scipy.stats import loguniform
import pandas as pd

**Lines 2-9** import our required Python packages. These imports are nearly identical to our previous two scripts, the most notable difference being `RandomizedSearchCV`, scikit-learn’s implementation of a randomized hyperparameter search.

With our imports taken care of we can load our abalone CSV file from disk and then preprocess it using the `StandardScaler`:

\# load the dataset, separate the features and labels, and perform a
# training and testing split using 85% of the data for training and
# 15% for evaluation
print("\[INFO\] loading data...")
dataset = pd.read\_csv(config.CSV\_PATH, names=config.COLS)
dataX = dataset\[dataset.columns\[:-1\]\]
dataY = dataset\[dataset.columns\[-1\]\]
(trainX, testX, trainY, testY) = train\_test\_split(dataX,
	dataY, random\_state=3, test\_size=0.15)

# standardize the feature values by computing the mean, subtracting
# the mean from the data points, and then dividing by the standard
# deviation
scaler = StandardScaler()
trainX = scaler.fit\_transform(trainX)
testX = scaler.transform(testX)

Next, we initialize our SVR model and then define the search space for our hyperparameters:

\# initialize model and define the space of the hyperparameters to
# perform the randomized-search over
model = SVR()
kernel = \["linear", "rbf", "sigmoid", "poly"\]
tolerance = loguniform(1e-6, 1e-3)
C = \[1, 1.5, 2, 2.5, 3\]
grid = dict(kernel=kernel, tol=tolerance, C=C)

From there we can tune the hyperparameters using a randomized search:

\# initialize a cross-validation fold and perform a randomized-search
# to tune the hyperparameters
print("\[INFO\] grid searching over the hyperparameters...")
cvFold = RepeatedKFold(n\_splits=10, n\_repeats=3, random\_state=1)
randomSearch = RandomizedSearchCV(estimator=model, n\_jobs=-1,
	cv=cvFold, param\_distributions=grid,
	scoring="neg\_mean\_squared\_error")
searchResults = randomSearch.fit(trainX, trainY)

# extract the best model and evaluate it
print("\[INFO\] evaluating...")
bestModel = searchResults.best\_estimator\_
print("R2: {:.2f}".format(bestModel.score(testX, testY)))

**Lines 40-42** instantiate our `RandomizedSearchCV` object, similar to how we created our `GridSearchCV` tuner.

From there, **Line 43** runs the randomized search over our hyperparameter space.

Finally, **Lines 47 and 48** grab the best model found during the hyperparameter space and evaluate it on our testing set.

### **Randomized hyperparameter search results**

We are now ready to see how our randomized hyperparameter search performs.

Access the **_“Downloads”_** section of this tutorial to retrieve the source code and Abalone dataset.

You can then execute the `train_svr_random.py` script.

$ time python train\_svr\_random.py
\[INFO\] loading data...
\[INFO\] grid searching over the hyperparameters...
\[INFO\] evaluating...
R2: 0.56

real	0m36.771s
user	4m36.760s
sys	0m6.132s

Here, we obtain a coefficient of determination value of `0.56` (the same as our grid search); however, the random search took only **36 seconds** instead of the grid search, which took **4m34s.**

**Whenever possible, I suggest you use a randomized search instead of a grid search for hyperparameter tuning.** You’ll typically obtain a similar accuracy in a fraction of the time.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial, you learned the basics of hyperparameter tuning using scikit-learn and Python.

We investigated hyperparameter tuning by:

1.  Obtaining a baseline accuracy on our dataset with _no hyperparameter tuning_ — this value became our score to beat
2.  Utilizing an exhaustive grid search
3.  Applying a randomized search

Both the grid search and random search beat the original baseline, but the random search did so in 86% less time.

I suggest using randomized searches when tuning your hyperparameters, as the time savings alone make it more practical and useful in machine learning projects.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch),** **_simply enter your email address in the form below!_**

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!