Hyperparameter tuning of Deep learning models in Keras
By: Krishna Prakash
Tags: Hyperparameter Tuning, Keras Tuner, Keras, Aisara Tuner, Deep Learning

KEYWORDS: Deep Learning, Model tuning, Hyperparameters, Hyperparameter tuning, Keras, Keras tuner, AiSara tuner, machine learning

Reading time: 10 mins approx.

Fig: Representing Images on Multiple Layers of Abstraction in Deep Learning ( Source & Acknowledgements to: Schulz, H., & Behnke, S. (2012). Deep Learning. KI — Künstliche Intelligenz, 26, 357–363 )

Machine learning model tuning is usually a trial-and-error process through in which we change some hyperparameters, run the model on the data again with the objective of improving the performance to determine which set of hyperparameters results in the most accurate model.

In my pursuit of building efficient deep learning models, the model tuning stage to get the best accuracy is what I found fascinating.

I learnt that accuracy-complexity balance depends on many hyperparameters and value of those parameters can have wide ranges. So, the challenge became clear. There is no single step that can give me one best solution. It involves several trials with hyperparameters and acceptable range of values.

So I decided to master the science of tuning of deep learning models. At first, I found a tuning approach using Keras Tuner. The example python script found in google colab was very useful. Soon I realized I have to go beyond Keras tuner as my goal was to design an efficient deep learning model. I have found another tool called AiSara Tuner for tuning deep learning model.

I have experimented with both the Keras tuners with various hyperparameters and benchmarked some measures to compare the tools. I have used fashion MNIST dataset and CIFAR-10 datasets for my experiments as it is convenient for anyone who wants to evaluate the tools from anywhere.

Metrics used for benchmark:

Experimental setup:

I have chosen a basic experimental dataset to enable all reviewers and learners to easily understand and verify the results by using commonly available dataset. So the scripts provided here can be executed from Google Colab by anyone from anywhere. All the experiments are run in Google Colab and shared in my GitHub repo here.

System environment

I have set Google Colab with Runtime hardware accelearator = “GPU” (Menu command Navigator: Runtime → Change runtime type)

Dataset

fashion MNIST dataset

CIFAR-10 dataset

Keras tuners

I have set up mirror environments to Keras tuner and AiSara tuner with separate python scripts.

Keras tuner

Source of the python installer package: https://pypi.org/project/keras-tuner/

AiSara tuner

Source of the python installer package: https://pypi.org/project/aisaratuners/1.1/

Hyperparameters:

I have used 3 hyperparameters number of layers, number of units, learning rate in my experiments.

Experimental Results

Below is the summary of all experimental results. You can refer to GitHub repo here to view the source and results obtained from test runs.

AiSara Keras tuner has excellent graphical visualization methods which can explain the search space. Few samples of visualizations generated by AiSara keras tuner shown below:

Graphical visualization of aisaratuner search space

Tabulated results of aisaratuner search space with optimization results

I hope this experiment will help all data science enthusiasts and deep learning buddies to understand the Keras tuners.

References

Schulz, H., & Behnke, S. (2012). Deep Learning. KI — Künstliche Intelligenz, 26, 357–363. Keras API: https://keras.io/api/

3. Keras Tuner : https://keras-team.github.io/keras-tuner/

4. AiSara Tuner: GitHub — aisara-hub/aisaratuners and https://pypi.org/project/aisaratuners/1.1/

5. Experimental python scripts: https://github.com/Krishnap641/Keras-tuner