Choosing a hyperparameter tuning library — ray[tune] or aisaratuners? | by Nouman | Towards Data Science

Creating a model is easy. What’s hard is building a model with optimal hyperparameters!. You can create a neural network with a random number of hidden layers and it will probably give you results better than random. But to get optimal results, you need to get the best hyperparameters that optimize the results. This process of finding the best hyperparameters is known as Hyperparameters tuning.
This is basically a time-consuming and a computationally expensive process as we have to search a pretty wide space in order to find these. And also we as humans, can’t try each and every one of the combinations and thus can’t claim a set of hyper parameters as the best ones. Enter hyper parameters tuning libraries. These libraries search the parameters space and calculate the metrics for each one. It lets you know the optimized hyper parameters for your model, and the best thing is you have to do minimal changes in the code.
Two such libraries, that I will be comparing in this article, are ray[tune] and aisaratuners. I will be testing them on the iris dataset to see which library performs better.
Spoiler Alert: If you just want to select one library from these, select aisaratuners. It outperforms ray[tune] by a large margin both in terms of accuracy and time
First, we will start with creating a model with random number of hidden layers to show how the model performs. Let’s create a function for creating a model.
After that, let’s create a function that loads the data, creates the model and also does the training. It will return the trained model and the history object containing details about the training metrics
This model accuracy will be quite low. In fact, it was able to achieve an accuracy of only 34.21% which is slightly above than random guessing.
Tune is a Python library for experiment execution and hyperparameter tuning at any scale. It’s core features are distributed hyperparameter tuning and automatic logging to Tensorboard. It lets you choose from a variety of algorithms for searching the hyperparameters space. You can see all of them here
Let’s start with installing the library
pip install -q -U ray[tune]
For ray[tune] to work, we first have to create a callback that can report at each epoch back to tune, so it can make better decision on how to choose the best hyperparameters.
After that, we have to create a function that accepts the object of hyperparameter space and goes through that to find the best parameters.
This function is called by the library and a hyperparameter space along with the number of samples (number of trials) are passed to it. The larger the number of trials, better the chances of finding the best hyper parameters.
The number of trials were set to 8 and the model was able to achieve an accuracy of 92.11%. Let’s see if aisaratuners can do it better.
Aisaratuners library is a hyperparameter tuning library for Machine learning models that aims to give you the best optimized model for your dataset. It uses AiSara propreitary state of the art algorithm. Aisaratuners uses Latin Hypercube Sampling to do the initial sampling of hyperparameters and then uses its HyperParameter tuning API which utlilizes SOTA pattern recognition so that the sampling space can be reduced in order to pick the best parameters for your model. It can be better explained by the following image from Aisara medium post.
Now let’s get hands-on with aisaratuners. We will start by installing and importing it.
!pip install aisaratunersfrom aisaratuners.aisara_keras_tuner import Hp, HpOptimization
We will start by first defining the hyperparameters and their space. I will set the same space as I did for ray[tune] for better comparison.
These hyperparameters are used in a function that tries values from this space to figure out the best possible values.
Finally, we have to run the optimizer.
So, Aisaratuners was able to achieve an accuracy of 97.37% in half as many trials (4 vs 8) as ray[tune].
This just shows that aisaratuners converges much faster due to its proprietary state-of-the-art algorithm for finding the best hyperparameters and also results in much better accuracy
The best thing I liked about aisaratuners is its simplicity as we had to do minimum changes which made sense intuitively as well and also that it gives the capability to visualize the process as well. Below are some visualizations about the optimization procedure
Lastly, I wanted to figure out if ray[tune] was able to achieve a higher accuracy if we tested with more trials. I doubled the number of trials from 8 to 16. This resulted in more time taken to find the best hyper parameters but it was able to achieve an accuracy of 97.37% in that case.
We can conclude the following from the experiments:
aisaratuners library was able to outperform ray[tune], both with respect to number of trials (4 versus 8) and test accuracy (97% versus 92%).ray[tune] was able to achieve equal accuracy as compared to aisaratuners but with four times as many trials (4 versus 16).There were less changes in code in aisaratuners as there were incase of ray[tune] i.e, it was much more easy to setup aisaratuners.aisaratuners gives access to an API to plot the optimization process to give a better understanding to the user.I personally did face some issues with ray[tune] when using it with Convolutional Layers.
aisaratuners library was able to outperform ray[tune], both with respect to number of trials (4 versus 8) and test accuracy (97% versus 92%).
ray[tune] was able to achieve equal accuracy as compared to aisaratuners but with four times as many trials (4 versus 16).
There were less changes in code in aisaratuners as there were incase of ray[tune] i.e, it was much more easy to setup aisaratuners.
aisaratuners gives access to an API to plot the optimization process to give a better understanding to the user.
I personally did face some issues with ray[tune] when using it with Convolutional Layers.
Feel free to reach out if you have any questions regarding the article. You can check the full code on my Google Colab here
If you feel that above content was useful for you, do share it and feel free to support me ->