Hyperparameter Optimization with the Keras Tuner, Part 4
By: David Patton
Tags: Hyperparameter Tuning, TensorFlow, Keras, Data Science, Deep Learning

Photo by Caleb Woods on Unsplash

In part 3 of this discussion, I introduced the concept of tuning hyperparameters which control the training of the model. I looked at learning rates and choices of optimizers. In this part I am going to introduce another of the built-in tuners in the Keras Tuner library and apply it to avoiding overfitting when training. Feel free to follow along here and in an example notebook on github.

Baseline

As in previous discussions, I ran a baseline model for comparison to the tuned models. Here is the baseline model summary.

Model: "functional_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= inp (InputLayer) [(None, 32, 32, 3)] 0 _________________________________________________________________ conv2d (Conv2D) (None, 30, 30, 16) 448 _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 15, 15, 16) 0 _________________________________________________________________ conv2d_1 (Conv2D) (None, 13, 13, 32) 4640 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 6, 6, 32) 0 _________________________________________________________________ conv2d_2 (Conv2D) (None, 4, 4, 64) 18496 _________________________________________________________________ max_pooling2d_2 (MaxPooling2 (None, 2, 2, 64) 0 _________________________________________________________________ flatten (Flatten) (None, 256) 0 _________________________________________________________________ dense (Dense) (None, 64) 16448 _________________________________________________________________ dense_1 (Dense) (None, 10) 650 ================================================================= Total params: 40,682

Trainable params: 40,682

Non-trainable params: 0

As show below I am going to train it with the Adam optimizer and default learning rate:

tf.keras.optimizers.Adam(learning_rate=1e-3)

After training this resulted in an accuracy of 66.5%. Note that I used filter sizes of 16, 32, and 64 in my Conv2d layers and 64 units in the first Dense layer.

The Hyperband Tuner

Next I built a hypermodel for tuning:

inp = Input(shape=(32, 32, 3), name='inp') x = Conv2D(hp.Int('f_1', 16, 96, 16), 3, activation='relu')(inp) x = MaxPool2D()(x) x = Conv2D(hp.Int('f_2', 32, 128, 32), 3, activation='relu')(x) x = MaxPool2D()(x) x = Conv2D(hp.Int('f_3', 64, 128, 32), 3, activation='relu')(x) x = MaxPool2D()(x) x = Flatten()(x) x = Dense(hp.Int('h_1', 64, 512, 64), activation='relu')(x) out = Dense(10, activation='softmax')(x) model = Model(inputs=[inp], outputs=[out])

I am tuning the size of the filters in the Conv2D layers and the hidden units in the first Dense layer. In order to tune these hyperparameters I am using the Hyperband Keras tuner:

tuner = kt.Hyperband(hypermodel=build_hypermodel,

objective='val_loss',

max_epochs=100,

factor=3,

hyperband_iterations=1,

directory='test_dir',

project_name='a',

seed=seed,

overwrite=True)

This tuner uses a bandit-based approach to optimization. More about this approach can be found at https://jmlr.org/papers/v18/16-558.html. The outcome of tuning was the optimal filter sizes are 32 , 128, and 128. The optimal hidden_units is 256. Using these hyperparameters in the model improves the accuracy to 69.7%.

OverFitting

Often models will overfit the training data and not generalize well to real world data (or at least the test data). Many deep learning approaches are available to remedy overfitting. However, they introduce even more hyperparameters. Fortunately we can use the Keras Tuner to help us choose them. I show a couple of them below:

Dropout

When adding a Dropout layer to a model there is a choice of the fraction of input units to drop. This “rate” is a value from 0 to 1. Here is an example of making that rate a tunable hyperparameter:

inp = Input(shape=(32, 32, 3), name='inp') x = Conv2D(48, 3, activation='relu')(inp) x = MaxPool2D()(x) x = Conv2D(96, 3, activation='relu')(x) x = MaxPool2D()(x) x = Conv2D(128, 3, activation='relu')(x) x = MaxPool2D()(x) x = Flatten()(x) dp = hp.Float('dp_1', min_value=0.0, max_value=0.5, step=0.1) x = Dropout(dp)(x) x = Dense(512, activation='relu')(x) out = Dense(10, activation='softmax')(x) model = Model(inputs=[inp], outputs=[out])

Note that the value is allowed to be from 0 to 0.5. If Dropout is not useful then 0.0 will be the best choice. One could of course use multiple Dropout layers in the model and tune them all.

Image Augmentation

Another popular approach for image classification problems is to use image augmentation to prevent overfitting. Here is a model that uses random image rotations as an augmentation approach. In it there is a hyperparameter to control the extent of the rotation (a fraction of 2Pi).

inp = Input(shape=(32, 32, 3), name='inp') f1 = hp.Float('f1', min_value=0.0, max_value=0.2, step=0.05) x = tf.keras.layers.experimental.preprocessing.RandomRotation(f1)(inp) x = tf.keras.layers.experimental.preprocessing.Rescaling(1.0 / 255)(x) x = Conv2D(48, 3, activation='relu')(inp) x = MaxPool2D()(x) x = Conv2D(112, 3, activation='relu')(x) x = MaxPool2D()(x) x = Conv2D(128, 3, activation='relu')(x) x = MaxPool2D()(x) x = Flatten()(x) x = Dropout(0.1)(x) x = Dense(512, activation='relu')(x) out = Dense(10, activation='softmax')(x) model = Model(inputs=[inp], outputs=[out])

The approach of using the augmentation as a layer of the model allowed me to tune the hyperparameter as part of the model. If so desired, after tuning I could use that tuned hyperparameter outside of the model by removing the layer and instead do the augmentation via a ImageDataGenerator or via a tf.data.DataSet map function.

Regardless, I believe this is a very powerful way to optimized the amount of augmentation necessary to produce the best model.

Tuning the tuners

Note that in this series I showed the implementation of the three built-in tuners provided by the Keras Tuner package. You can implement you own as well and I suspect many may show up on github in the future for plug-n-play utility.

One warning is that with each tuner there are many more knobs to turn. For example, how many epochs to run the for, etc. Now we just need to do hyperparameter tuning of the tuners themselves based on outcomes and our budget!

I hope you enjoyed this series and as always the code is available here: https://github.com/dcpatton/keras_tuner/blob/master/keras_tuning_part4.ipynb