Hyperparameter Tuning with Keras Tuner | by Cedric Conol | Towards Data Science

Great data scientists do not settle with “okay”, they go beyond to achieve the extraordinary.
In this article, we’ll review techniques data scientists use to create models that work great and win competitions. Getting the most out of our models means choosing the optimal hyperparameters for our learning algorithm. This task is known as hyperparameter optimization or hyperparameter tuning. This is especially strenuous in deep learning as neural networks are full of hyperparameters. I’ll assume that you are already familiar with common data science concepts like regression and mean squared error (MSE) metric and have experience building model using tensorflow and keras.
To demonstrate hyperparameter tuning methods, we’ll use keras tuner library to tune a regression model on the Boston housing price dataset. This dataset contains 13 attributes with 404 and 102 training and testing samples respectively. We’ll use tensorflow as keras backend so make sure you have tensorflow installed on your machines. I’m using tensorflow version ‘2.1.0’ and kerastuner version ‘1.0.1’. Tensorflow 2.0.x comes up with keras so you don’t need to install keras separately if you have version 2.0.x. You can check the version you have using the code below:
import tensorflow as tfimport kerastuner as ktprint(tf.__version__)print(kt.__version__)
Boston housing price regression dataset can be downloaded directly using keras. Here’s a list of datasets that comes with keras. To load the dataset, run the following codes.
from tensorflow.keras.datasets import boston_housing(x_train, y_train), (x_test, y_test) = boston_housing.load_data()
Note that if this is the first time you are using this dataset within keras, it will download the dataset from an external source.
This is the regression model I’ll use in this demo. The code below shows how the model was built without any tuning.
from sklearn.preprocessing import StandardScalerfrom tensorflow.keras import models, layers# set random seedfrom numpy.random import seedseed(42)import tensorflowtensorflow.random.set_seed(42)# preprocessing - normalizationscaler = StandardScaler()scaler.fit(x_train)x_train_scaled = scaler.transform(x_train)x_test_scaled = scaler.transform(x_test)# model buildingmodel = models.Sequential()model.add(layers.Dense(8, activation='relu', input_shape=(x_train.shape[1],)))model.add(layers.Dense(16, activation='relu'))model.add(layers.Dropout(0.1))model.add(layers.Dense(1))# compile model using rmspropmodel.compile(optimizer='rmsprop',loss='mse',metrics=['mse'])# model traininghistory = model.fit(x_train_scaled, y_train, validation_split=0.2, epochs=10)# model evaluationmodel.evaluate(x_test_scaled, y_test)
This model has a MSE of around 434. I have set the random seed in numpy and tensorflow to 42 to get reproducible results. Despite doing so, I still get slightly different results every time I run the code. Let me know in the comments what else I missed make this reproducible.
To start tuning the model in keras tuner, let’s define a hypermodel first. Hypermodel is a keras tuner class that lets you define the model with a searchable space and build it.
Create a class that inherits from kerastuner.HyperModel, like so:
from kerastuner import HyperModelclass RegressionHyperModel(HyperModel):    def __init__(self, input_shape):        self.input_shape = input_shape    def build(self, hp):        model = Sequential()        model.add(            layers.Dense(                units=hp.Int('units', 8, 64, 4, default=8),                activation=hp.Choice(                    'dense_activation',                    values=['relu', 'tanh', 'sigmoid'],                    default='relu'),                input_shape=input_shape            )        )                model.add(            layers.Dense(                units=hp.Int('units', 16, 64, 4, default=16),                activation=hp.Choice(                    'dense_activation',                    values=['relu', 'tanh', 'sigmoid'],                    default='relu')            )        )                model.add(            layers.Dropout(                hp.Float(                    'dropout',                    min_value=0.0,                    max_value=0.1,                    default=0.005,                    step=0.01)            )        )                model.add(layers.Dense(1))                model.compile(            optimizer='rmsprop',loss='mse',metrics=['mse']        )                return model
This is the same model we built earlier, except that for every hyperparameter, we defined a search space. You may have noticed hp.Int, hp.Float, and hp.Choice, these are used to define a search space for a hyperparameter that accepts an integer, float and a category respectively. A complete list of hyperparameter methods can be found here. ‘hp’ is an alias for Keras Tuner’s HyperParameters class.
Hyperparameter such as the number of units in a dense layer accepts an integer, hence, hp.Int is used to define a range of integers to try. Similarly, the dropout rate accepts a float value so hp.Float is used. Both hp.Int and hp.Float requires a name, minimum value and maximum value, while the step size and default value is optional.
The hp.Int search space below is named, “units”, and will have values from 8 to 64 in multiples of 4, and a default value of 8. hp. Float is used similarly as hp.Int but accepts float values.
hp.Int('units', 8, 64, 4, default=8)
hp.Choice is used to define a categorical hyperparameter such as the activation function. The search space below, named “dense_activation”, will choose between “relu”, “tanh”, and “sigmoid” functions, with a default value set to “relu”.
hp.Choice('dense_activation', values=['relu', 'tanh', 'sigmoid'], default='relu')
Let’s instantiate a hypermodel object. Input shape varies per dataset and the problem you are trying to solve.
input_shape = (x_train.shape[1],)hypermodel = RegressionHyperModel(input_shape)
Let’s start tuning!
As the name suggests, this hyperparameter tuning method randomly tries a combination of hyperparameters from a given search space. To use this method in keras tuner, let’s define a tuner using one of the available Tuners. Here’s a full list of Tuners.
tuner_rs = RandomSearch(            hypermodel,            objective='mse',            seed=42,            max_trials=10,            executions_per_trial=2)
Run the random search tuner using the search method.
tuner_rs.search(x_train_scaled, y_train, epochs=10, validation_split=0.2, verbose=0)
Select the best combination of hyperparameters the tuner had tried and evaluate.
best_model = tuner_rs.get_best_models(num_models=1)[0]loss, mse = best_model.evaluate(x_test_scaled, y_test)
Random search’s MSE is 53.48, a very big improvement from not performing any tuning at all.
Hyperband is based on the algorithm by Li et. al. It optimizes random search method through adaptive resource allocation and early-stopping. Hyperband first runs random hyperparameter configurations for one iteration or two, then selects which configurations perform well, then continues tuning the best performers.
tuner_hb = Hyperband(            hypermodel,            max_epochs=5,            objective='mse',            seed=42,            executions_per_trial=2        )tuner_hb.search(x_train_scaled, y_train, epochs=10, validation_split=0.2, verbose=0)best_model = tuner_hb.get_best_models(num_models=1)[0]best_model.evaluate(x_test_scaled, y_test)
The resulting MSE is 395.19 which is a lot worse when compared to random search but a little bit better than not tuning at all.
Bayesian optimization is a probabilistic model that maps the hyperparameters to a probability score on the objective function. Unlike Random Search and Hyperband models, Bayesian Optimization keeps track of its past evaluation results and uses it to build the probability model.
tuner_bo = BayesianOptimization(            hypermodel,            objective='mse',            max_trials=10,            seed=42,            executions_per_trial=2        )tuner_bo.search(x_train_scaled, y_train, epochs=10, validation_split=0.2, verbose=0)best_model = tuner_bo.get_best_models(num_models=1)[0]best_model.evaluate(x_test_scaled, y_test)
Best model MSE tuned using Bayesian optimization is 46.47, better than the first two tuners we have tried.
We were able to show that indeed, tuning helps us get the most out of our models. Discussed here are just 3 of the many methods of hyperparameter tuning. When trying out the codes above, we may get slightly different results, for some reason, despite setting numpy, tensorflow, and keras tuner random seeds, results per iteration still differ slightly. The notebook is uploaded in my github repo.
Furthermore, tuners can also be tuned! Yes, you read that right, tuning the tuners. Tuners accept values such as max_trials and execution per trial and are can, therefore, be tuned as well. Try changing these parameters and see if you get further improvements.
[1] F. Chollet, Deep Learning with Python (2018), Manning Publications Inc.
[2] Keras Tuner Documentation, https://keras-team.github.io/keras-tuner/
[3] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, A. Talwalkar, Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2018), https://arxiv.org/abs/1603.06560