Part 2: Hyperopt.. In this blog series, I am comparing... | by Jakub Czakon | Towards Data Science

In this blog series, I am comparing python HPO libraries. Before reading this post, I would highly advise that you read Part 0: Introduction where I:
talked about what HPO is,
selected libraries to compare,
selected the evaluation criteria,
defined an example problem for the HPO.
Code for this blog post and other parts of the series is available on github while all the experiments with scripts, hyperparameters, charts, and results (that you can download) are available for you in Neptune. For more information, read this post about organizing the experimentation process.
Ok, let’s do this.
This is the classic in the HPO space. This project has over 3300 stars, 600 forks and 40 contributors(2 main ones). There are even projects built on top of it like:
hyperas: hyperopt + keras
hyperopt-sklearn: hyperopt + sklearn
The API is pretty simple and easy to use. We need to define a search space, objective and run the optimization function:
First, define the search space. It can be a regular dictionary or as in this case OrderedDict:
What is kind of awkward is that I have to define the name twice, once as a dictionary key and once as a label in the parameter sampler.
Then, define the objective function. Remember that hyperopt minimizes the function, so change the sign if you need to:
Finally, instantiate the Trials object, and run the optimization:
...and done!
All the information, hyperparameters, loss and other things are kept inside the trialsobject. You can access it with no problems or pickle it for later. If you want to see a full example script with training curves, diagnostic charts and all that, see this versioned ml experiment.
I give it a score of 9/10 because of the name duplication in the space search definition and the fact that for simple problems I would rather not have to instantiate thetrialsobject but simply have it instantiated asTrials() by default.
Score 9/10
Search Space
This is where hyperopt shines. There is a ton of sampling options to choose from:
Categorical parameters-use hp.choice
Integer parameters-you can use hp.randit, hp.quniform, hp.qloguniform or hp.qlognormal which really gives you a lot of options to model your integer hyperparameter space
Float parameters- similarly to integer parameters you really get to choose what works for your problem with hp.normal, hp.uniform, hp.lognormal and hp.loguniform
But that is not it. You can easily define nested spaces where values of some parameters depend on others. Taking the example from the documentation:
we can see that by combining hp.choice with other sampling methods we can have conditional search spaces.
This is very useful when searching over complicated spaces with hyperparameters spanning from your feature preprocessing, through model selection methods all the way to model hyperparameters.
Optimization methods
There are two optimization algorithms to try.
tpe.rand.suggest
Your standard random search over the parameters.
tpe.suggest
Similarly to what was explained in the previous blog post about Scikit-Optimize we want to use a cheap surrogate model that estimates the performance of the expensive objective function on a set of parameters. The difference is that instead of estimating the actual performance (point estimation) we want to estimate the density in the tails.
I like the following explanation taken from the AutoML_Book by amazing folks over at AutoML.org Freiburg.
Instead of modeling the probability p(y|λ) of observations y given the configurations λ, the Tree Parzen Estimator models density functions p(λ|y < α) and p(λ|y ≥ α). Given a percentile α (usually set to 15%), the observations are divided in good observations and bad observations and simple 1-d Parzen windows are used to model the two distributions.
By using p(λ|y < α) and p(λ|y ≥ α) one can estimate the Expected Improvement (EI) of a given run.
Interestingly there are no options to specify the λ parameter in the optimizer.
Callbacks
There are no callbacks per se, but you can put your callback function inside the objective and it will be executed every time the objective is called.
I don’t love it but I guess I can live with that.
Summing up, a big plus for the nested search space but I don’t like the fact that there are no clean callbacks. TPE doesn’t have any (hyper)hyperparameters which on one hand is great (less work, less overfitting) but on the other hand, it doesn’t allow you to adjust things when they are not working. I give it 7/10.
Score 7/10
Putting it mildly, it could use a lot of love.
The official documentation page is hosted on wiki. It is more of a library tour than proper documentation if you ask me. That being said, having read it, you should be able to use this library effectively. The problem is, when you want to find one particular thing, you need to go through the entire wiki looking for it.
It is missing basic examples. The only one I found (hyperopt/recipes) is 6 years old and with errors.
The vast majority of functions and main class methods are missing docstrings. Unfortunately, it makes it difficult to navigate and answer simple questions like what does this function actually do?
The good thing is, there are a lot of blog posts about it. Some of them that I found useful are:
“Parameter Tuning with Hyperopt” by District Data Labs
“Hyperopt tutorial for Optimizing Neural Networks’ Hyperparameters” by Vooban
“On Using Hyperopt: Advanced Machine Learning” by Tanay Agrawal
“An Introductory Example of Bayesian Optimization in Python with Hyperopt” by Will Koehrsen
The documentation is definitely not a strong side of this project but because it’s a classic there are a lot of outside resources. I give it 3/10.
Score 3/10
Note
If you are thinking about doing something good for the community, you can go ahead, add docstrings, a bunch of examples, and create read-the-docs documentation for this project. I, for one, would be grateful.
hyperopt has a visualization module plotting.py. It has three functions:
main_plot_history -it shows you the results of each iteration and highlights the best score.
main_plot_histogram -shows you the histogram of results over all iterations.
main_plot_vars -which... I don’t really know what it does as I couldn’t get it to run and there were no docstrings nor examples (again, the documentation is far from perfect).
There are some basic visualization utilities but they are not super useful. Feels like 3/10.
Score 3/10
Note
As I mentioned in my last post I like the plots that come with Scikit-Optimize and I implemented a set of functions that help with conversions between different HPO libraries so that you could use those visualizations for every lib. I’ve put them in the neptune-contribpackage and you can check how to use it here.
You can just pickle the trialsobject and use it later. For example, the saving part could read:
Loading again is done by pickle while restarting is as simple as passing the trials object (now not empty) and changing the max_evals from 100 to 200.
Simple and works with no problems: 10/10.
Score 10/10
With hyperopt, you can distribute your computation over a cluster of machines. Good step-by-step instructions can be found in this blog post by Tanay Agrawal but in a nutshell, you need to:
Start a server with MongoDB on it which will consume results from your worker training scripts and send out the next parameter set to try,
In your training script, instead of Trials()create a MongoTrials object pointing to the database server you have started in the previous step,
Move your objective function to a separate objective.py script and rename it to function,
Compile your Python training script,
Run hyperopt-mongo-worker
Though it gets the job done it doesn’t feel quite perfect. You need to do some juggling around the objective function, and starting MongoDB could have been provided in the CLI to makes things easier.
Because of the above 8/10 feels fair to me.
Score 8/10
You can explore all the versioned ml experiments in Neptune. Every experiment has a script attached to it. For example, you can see the code for the best one here.
You can also download experiment metadata to pandas.DataFrame by running:
Let’s take a look at the experiment results:
Thetpemethod was a slight winner but only when we run it for more iterations. At the predefined budget of 100 iterations randomsearch has slightly outperformed tpe .
Looking at the plot_evaluations chart from Scikit-Optimize for the best 100 iterations experiment:
we can see that only for the feature_fraction parameter the search has actually steered in some direction. Otherwise, it was pretty much random all the way( which explains why the results are so close to random search).
Overall the highest score I got for 100 iteration runs was merely 0.84509 which was actually worse than a random search (from Scikit-Optimize) 0.8464 by ~0.001. I will translate that to -1 point (-0.001*100).
Score –1
Hyperopt is a classic and it has served the community for many years. It has an easy to use API, can parallelize computations on a cluster of machines, and has really great support for nested search spaces.
However, it’s limited documentation with little examples and docstrings can make it hard to use, especially outside of the golden path. Visualization suite is quite limited as well and doesn’t bring a lot of value. What is a bit troubling, is the fact that for our example problem it couldn’t beat random search.
Let’s take a look at the results for all of the criteria:
The score of 39 is way lower than that of Scikit-Optimize. The big factors were documentation, visualizations and experimental results.
Next time we will take a close look at another Tree-Parzen estimator library for hyperparameter optimization Optuna.
You can check out part 3 of the series Optuna vs Hyperopt here.
You can also go and read other posts from this series in my HPO in Python column.
If you liked this, you can find more posts like this on our Neptune blog.
You can also find me tweeting @Neptune_ai or posting on LinkedIn about ML and Data Science stuff.
@book{automl_book,    editor = {Hutter, Frank and Kotthoff, Lars and Vanschoren, Joaquin},    note = {In press, available at http://automl.org/book.},    publisher = {Springer},    title = {Automated Machine Learning: Methods, Systems, Challenges},    year = {2018}}