Compare popular AutoML frameworks on 10 tabular Kaggle competitions | by Piotr Płoński | Towards Data Science

Have you ever lost your evening juggling different many Machine Learning algorithms and packages, switching hyperparameters values and trying different preprocessing methods? Building a good Machine Learning pipeline requires a lot of try-and-error work. Thus, it’s very time-consuming and can be exhausting.
The Automated Machine Learning (AutoML) frameworks aim to help.
I started to work on AutoML systems in 2016. I was a Ph.D. student working on many ML projects from different domains: high-energy physics, medicine, and bioinformatics. I needed elegant software that I can use to swap my code with lots of tuning and validation loops. This is how I started to work on the project that I called the MLJAR. The name comes from ML + JAR, but the JAR here has the second meaning (I was studying in Warsaw, Poland):
JAR is an informal term used for incoming residents of Warsaw. The term is derived from jars which are used by many students and workers for carrying food from their parent’s house when visiting during the weekend.
You can read more about JARs in this article. It has a rather negative meaning. I thought that I will use it to name my new project. I will make the project successful to show that it is a mistake to treat newcomers badly.
The MLJAR AutoML engine is available as an open-source with code available on GitHub: https://github.com/mljar/mljar-supervised
The MLJAR AutoML works with tabular datasets.
The biggest advantage of the MLJAR AutoML is its transparency. There is no black-box magic in it. You can check the details of every step of the AutoML training. All information about trained models is saved in the hard drive (auto-save is switched on, always). Each model has its own README.md file. So you can easily check details (for example in GitHub, VS Code, or Jupyter Notebook). Below the example of a Markdown README.md report from the model training:
In the past, AutoML was only for hyper-parameters optimization. It is the simplest thing that can be automated in the ML pipeline. Today, AutoML can serve many purposes. The MLJAR AutoML can work in three modes:
Explain — It is fast. Takes only a few minutes. Should be used to get you familiar with a new dataset. It does Exploratory Data Analysis. It trains only a few models with default hyper-parameters (no tuning!). For each model, a full set of explanations are produced: feature importance, SHAP dependency plots, decision plots, learning curves, and more.
Compete — The highly-tuned ML pipeline. Should be used for building high-accuracy models. The training can take long, easily more than 4 hours (depending on the dataset). It is using advanced feature engineering, ensembling, and stacking.
Perform — The mode for training production-ready ML models. The balance between speed of training and final accuracy.
Selection of the training mode is as simple as setting on parameter:
# initialization of AutoML with Compete mode# mode can be: Explain, Compete, Performautoml = AutoML(mode="Compete")
Feature engineering is very important for the performance of the Machine Learning pipeline. The MLJAR AutoML provides advanced feature engineering methods. It can generate new features with K-Means or Golden Features Search. There is also a Feature Selection procedure that can work with any Machine Learning algorithm.
Let’s check the AutoML performance on the most challenging datasets. The Kaggle platform hosts Data Science competitions with real-life data problems. I’ve used 10 tabular datasets from Kaggle that represent various Machine Learning tasks:
binary classification
multi-class classification
regression
Datasets selection is the same as in the article: N. Erickson, et al.: AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (Except House Prices Adv. Regression, because in the article there were no results from Private Leaderboard, the competition was still ongoing).
There were evaluated following frameworks:
AutoWeka
Auto-Sklearn
TPOT
H2O
Google Cloud AutoML Tables
AutoGluon
MLJAR
The frameworks were trained on m5.24xlarge EC2 machines (96CPU, 384 GB RAM). The training time was set to 4 hours. (Except GCP-Tables which was using its own machine types)
The final results presented below are presented as Percentile Rank in the Private Leaderboard (evaluated internally by Kaggle). The higher the value, the better. The 1st place solution in the Kaggle competition will get Percentile Rank equal 1.0. The last place in the solution in the competition will give 0 value. The results for AutoMLs other than MLJAR are from AutoGluon paper.
You can see that some AutoML frameworks jump into the Top-10% of the competition (without any human help)! The MLJAR AutoML without any human intervention was 5 times in the Top-25% out of the 10 Kaggle competitions. What is more, it was 3 times in the Top-10%.
Nowadays AutoML is a multi-purpose tool. Can be used by:
software engineers that need to apply ML in the application but don't know details how to tweak ML algorithms
citizen data scientists to build ML pipeline in a low-code environment
data scientists and engineers to speed-up their work
I hope that many people will benefit from my AutoML system. I put a lot of effort into it. I’m very interested in your opinion and feedback about MLJAR AutoML, and AutoML in general.
You can find MLJAR AutoML at GitHub: https://github.com/mljar/mljar-supervised