Picking a Hyperparameter Tuning Library for Your Model
By: Osama Akhtar
Tags: Hyperparameter Tuning, Keras, Machine Learning, Data Science, Deep Learning

Enough talking, let’s test the library to see if it actually gets the job done.

We will use the Fashion MNIST dataset since it is easy to work with and is still slightly difficult to get 90%+ validation accuracy. To keep this article concise, I will only include code relevant to the library and not include the code for loading and preprocessing data. You can refer to the notebooks at the end of this article.

We will train a rather simple but dynamic model, with the help of aisaratuners library. Here we go.

To get started, we need to pip install the package:

!pip install aisaratuners # pip install using a jupyter notebook

We only need two imports from aisaratuners library to get started, yes, it is that simple!

Now comes the fun part, we get to decide which hyperparameters we get to tune dynamically for our model.

We instantiate our Hp object which is a container for all the hyperparameters we wish to tune. You can set min/max ranges, as well.

Next, in a function containing our model, we will include the Hp object and all the ranges we have defined:

One thing I like while working with aisaratuners was the fact that I had to introduce minimal and understandable changes to the model. We just need to use the Hp object we created and access any random number we set in the numrange method in the above cell.

Let’s now run the optimizer and see how the algorithm performs:

We have set 3 rounds, each with 5 trials. This means in any round, the algorithm will sample 5 sets of hyperparameters from the solution space and train models. This is depicted in Fig. 1 above. In each round, the model will try to reduce this solution space. Also, we have set two criteria, one to maximize the validation accuracy and one to minimize validation loss. The mode argument is set to ‘p’ to use the free available API for aisaratuners to function. You could use a commercial API to use as well. Find more information here.

It took me 889.84 seconds to run this optimization.

Please note that this time will vary for everyone and I was running the optimization using Google Colab, which has a variety of GPUs.

It took me less than 15 minutes (14.81) minutes to acquire the best model. That is amazing. It usually would take me at least two hours of manual tuning. Instead, we automatically tuned for the best model using aisaratuners in much less time.

Here are the results:

Fig. 2: Round 1: Orange, Round 2: Green, Round 3: Yellow

The five trials of each round are labeled. My findings were:

Orange: Some variance (e.g. 48% validation accuracy) in the first round, but overall, the accuracies were definitely in the higher region. This is impressive! Aisaratuners algorithm did well in initial sampling. Green: In the second round, the search space was reduced well enough because all the models were now achieving 90%+ validation accuracies. Yellow: Even now, when typically I would stop trying to achieve better results, aisaratuners still managed to squeeze a one per cent increase on the validation set.

This was exactly what I needed: an automatic hyperparameter tuner. What exceeded my expectations were the visualizations this library provided me with. The above information can be visualized in a picture-worth-a-thousand-words manner. Interactive visualizations are labeled in the captions.

(Interactive) Infer how in each round the validation accuracy is increasing.