Tune the hyperparameters of your deep learning networks in Python using Keras and Talos
By: Yefeng Xia
Tags: Data Science, Programming, Computer Vision, Python, Deep Learning

With the development of Deep Learning frameworks, it’s more convenient and easy for many people to design the architecture for an artificial neural network. The 3 most popular frameworks, Tensorflow, Keras, and Pytorch, are used more frequently. To improve the performance of our neural networks, there are many approaches, e.g. improve the data quality, using data augmentation. However, data quality is the source of data science. To get better data quality is usually extra expensive, time- and human resource-consuming. Therewith, we prefer to handle the hyperparameters/parameters of a model🏄🏼.

Let’s start it!

1. Parameters or Hyperparameters

A model parameter is a configuration variable that is internal to the model. It’s dependent on the model’s training data. The parameter of a model can be estimated by fitting the given data to the model.

A model hyperparameter is a configuration that is external to the model. The hyperparameters are to help us find model parameters, which are not dependent on training data.

On the contrary, we find the model’s parameter by setting a series of hyperparameters properly, which optimize the training process and make the best use of data. It means that we can manually give hyperparameters values and they can’t be updated during the training. But parameters are something instinct of the model, which updates continuously during the training.

An inappropriate analogy, if we regard a student as a model, his knowledge, characters, skills are more like the model’s parameter. The way we train him to get these abilities and features can be treated as hyperparameters.

Since Hyperparameters are the key to the model’s parameters, we should pay a lot of attention to them. How to select the model’s hyperparameters? To deal with the question requires enough knowledge and patience.

2. Strategies to tune hyperparameters

There are typically 5 different optimization techniques:

Manual Search: we choose some model hyperparameters based on our judgment/experience. We then train the model, evaluate its accuracy and start the process again. This loop is repeated until a satisfactory accuracy is scored. Grid search: a grid of hyperparameters and train/test our model on each of the possible combinations over a given subset of the hyperparameters space of the training algorithm. It’s the traditional method of hyperparameters optimization. Random Search: it overrides the complete selection of all combinations by their random selection. Therewith, it can reduce the number of search iterations by selecting some random combination of these hyperparameters. Bayesian Optimization: is a sequential design strategy for global optimization of black-box functions. It reduces the number of search iterations by choosing the input values bearing in mind the past outcomes. Evolutionary Algorithm: create a population of N Machine Learning models with some predefined Hyperparameters. It generates some offsprings having similar Hyperparameters to the ones of the best models so that to get again a population of N models. Just the best models will survive at the end of the process by sequentially selecting, combining, and varying parameters using mechanisms that resemble biological evolution. It simulates the process of natural selection which means those species who can adapt to changes in their environment can survive and reproduce and go to the next generation.

3. our own approach: grid search

In our work, we used often grid search. Grid search suffers from high dimensional spaces, but often can easily be parallelized, since the hyperparameter values that the algorithm works with are usually independent of each other. Besides, we write the code on the platform Colab, which allows us to write and execute Python in your browser:

Zero configuration required

Free access to GPUs

Easy sharing

4. Keras and Talos

If you want to quickly build and test a neural network with minimal lines of code, then Keras is what you need. Keras is an open-source neural network library written in Python that is an API designed for human beings, not machines. Since Tensorflow 2 comes up with a tight integration of Keras and an intuitive high-level API tf. keras, there are 2 ways to use Keras, either directly import Keras or from tf import Keras.

Talos was released on May 11, 2018 and has since been upgraded seven times. When running the code with Talos in the scan-command, all possible combinations are tested in an experiment.

Important: Talos radically changes the ordinary Keras workflow by fully automating hyperparameter tuning and model evaluation. Talos exposes Keras functionality entirely and there is no new syntax or templates to learn.

we can install talos with one line of command:

pip install talos

5. Dogs vs. Cats classifier with CNN

To make our result visible and intuitive, we take a simple case to classify whether images contain either a dog or a cat with CNN, the ancient problem in Computer Vision😆. I downloaded the image dataset from Kaggle. The dataset is in the ZIP file format after you download it from the link.

Photo by Hanna Listek on Unsplash

the unzipped dataset in Colab

from google.colab import drive drive.mount('/content/gdrive/') !mkdir -p dataset !unzip /content/gdrive/My\ Drive/Colab\ Notebooks/blogs_medium/cat_dog.zip -d dataset/

We can use a few lines of code to unzip the file directly in Google Colab.

Here we use LeNet-5, which is the 22-year-old neural network, usually as a teaching sample.