AutoML in Python: A comparison between Hyperopt Sklearn and TPOT | by Angelica Lo Duca | Towards Data Science

Automated Machine Learning (AutoML) involves the automation of the tasks of applying Machine Learning to real-world problems. AutoML covers the complete pipeline from the raw dataset to the deployable machine learning model.
Python provides some libraries which provide AutoML. In this tutorial I compare two popular libraries: Hyperopt Sklearn and TPOT. Another library exists, called AutoSklearn, which has not been tested in this article, because it is not compatible with some operating systems.
Both the analysed libraries are compatible with scikit-learn, the famous Python library for Machine Learning.
The complete code of this tutorial can be downloaded from my Github repository.
As use case for this tutorial, I exploit the heart.csv dataset, provided by the Kaggle repository. The dataset regards heart attacks and contains 14 input features and provides a binary classification as output (heart attack yes or no).
This library can be easily installed, as explained in the documentation and it is very simple to use. The documentation is simple but clear. It supports many scikit-learn classifiers, regressors and preprocessing models. In order to build an AutoML model, it is sufficient to create an instance of the HyperoptEstimator() , configure some parameters (e.g. type of estimation) and then fit the model. The bestmodel()of the HyperoptEstimatorreturns an object containing the full scikit-learnpipeline.
Let us suppose that X and y contain the input features and the targets respectively. We can split them into training and test sets through the scikit-learn function train_test_split():
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
Now we can train the HyperoptEstimator to search for the best classifier:
from hpsklearn import HyperoptEstimatorestim = HyperoptEstimator()estim.fit( X_train, y_train )
and we can get the best model:
best_model = estim.best_model()
which gives the following output:
{'learner': ExtraTreesClassifier(criterion='entropy', max_features='sqrt', n_estimators=13,                      n_jobs=1, random_state=4, verbose=False), 'preprocs': (), 'ex_preprocs': ()}
The key/value pair, learner contains the best model, the second one (preproc) the preprocessing techniques applied. In this case it is empty.
The Hyperopt Sklearn library reflects the stochastic nature of Machine Learning: in fact, if we try to run again the classifier over the same training set, we obtain another result.
Running the same code with the same training set (without performing again training/test split), we obtain the following best model:
{'learner': GradientBoostingClassifier(learning_rate=0.00813591154617836, max_depth=None, max_features=0.6562885234780235, min_samples_leaf=11, n_estimators=63, random_state=0, subsample=0.68797222866341), 'preprocs': (MinMaxScaler(feature_range=(-1.0, 1.0)),), 'ex_preprocs': ()}
Anyway, we can calculate the performance of the model. We can exploit the builtin function score() or define our performance metrics. In this last case, firstly, we must apply the obtained preprocessing (if any) to the test set and compare predicted results with actual values:
X_test_norm = X_testif len(best_model['preprocs']) > 0:    preprocs = best_model['preprocs'][0]    X_test_norm = preprocs.transform(X_test)
The Hyperopt Sklearn library does not provide the predict_proba() function, thus we must calculate it manually:
model = best_model['learner']y_score_hp = model.predict_proba(X_test_norm)
Then, we can plot the ROC curve and the precision recall curve provided by the scikitplot library, as follows:
import matplotlib.pyplot as pltfrom sklearn.metrics import roc_curvefrom scikitplot.metrics import plot_roc,aucfrom scikitplot.metrics import plot_precision_recall# Plot metrics plot_roc(y_test, y_score_hp)plt.show()    plot_precision_recall(y_test, y_score_hp)plt.show()
We can also calculate precision, recall and accuracy:
y_test_pred = model.predict(X_test_norm)accuracy = accuracy_score(y_test, y_test_pred)precision = precision_score(y_test, y_test_pred)recall = recall_score(y_test, y_test_pred)print('accuracy:  %s ' %(round(accuracy, 2)))print('precision:  %s ' %(round(precision, 2)))print('recall:  %s ' %(round(recall, 2)))
which gives the following output:
accuracy:  0.85 precision:  0.85 recall:  0.9
In the full code available in my Github repository, I also calculate the precision, recall and accuracy of the obtained model through 10-Fold validation. The following figure shows the precision trend over the number of folds:
I note that the model is not overfitted.
TPOT is another Python library for AutoML. Its documentation is well done. However, it is not simple to install it, because it is built on the top of other libraries, thus you must install firstly them to get TPOT working and running. Personally, I encountered many problems before installing it correctly.
Once installed, we can create an estimator, either a TPOTClassifier() or a TPOTRegressor(). In our case, we build a TPOT classifier:
from tpot import TPOTClassifiertpot = TPOTClassifier(generations=5, population_size=50, random_state=42)tpot.fit(X_train, y_train)
The process is very slow and takes a while before finishing. We can calculate, precision, recall and accuracy of the model:
y_test_pred = tpot.predict(X_test)accuracy = accuracy_score(y_test, y_test_pred)precision = precision_score(y_test, y_test_pred)recall = recall_score(y_test, y_test_pred)print('accuracy:  %s ' %(round(accuracy, 2)))print('precision:  %s ' %(round(precision, 2)))print('recall:  %s ' %(round(recall, 2)))
which gives the following output:
accuracy:  0.8 precision:  0.84 recall:  0.81
With respect to the best model obtained through Hyperopt Sklearn, TPOT behaves worse.
TPOT provides also the predict_proba() method, which can be used to calculate the ROC curve and the precision recall curve.
We can access the best model as a scikit-learnpipeline:
tpot.fitted_pipeline_
which gives the following output:
Pipeline(steps=[('logisticregression',                 LogisticRegression(C=20.0, random_state=42))])
Although in the documentation there is said that the model may produce different best models on the same dataset, in our case, running more times the same code on the same dataset returns the same model.
Now, I can calculate the precision, recall and accuracy through 10-Fold cross validation. TPOT provides cross validation natively. However, in this case, we calculate it manually, because we want to plot the trend of metrics versus each fold.
The following figure shows the precision trend. For the other plots (recall and accuracy), please check my Github repository:
With respect to the Hyperopt Sklearn library, TPOT seems more stables with different datasets.
In this article, we have compared two popular Python library for AutoML: Hyperopt Sklearn and TPOT.
Hyperopt Sklearn
Pros
easy to install
fast
the best model reaches high performance
Cons
different best models for the same (simple) dataset
simple and poor documentation
TPOT
Pros
always the same best model for the same (simple) dataset
well written documentation
the best model reaches good performance
Cons
very slow
difficult to install
If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and and Github.