AutoML is Overhyped. AutoML as an idea of automating a... | by Denis Vorotyntsev | Towards Data Science

I used AutoML in my work, in several ML competitions for blending with my main models and participated in two AutoML competitions. I think AutoML as an idea of automating a modeling process is excellent, yet the field is too overhyped. Some key concepts, like features engineering or meta-learning for hyperparameters optimization, will unleash its potential, but for now, boxed AutoML as a tool is just a waste of money.
Any Data Science project consists of several essential steps: formulation the problem from a business perspective (selecting the task and metric of success), gathering the data (collecting, cleaning, exploring), building a model and evaluating its performance, deploying model in production and observing model’s performance in production.
Each part of the process is vital for the project success. However, from the perspective of Machine Learning adept, the modeling part is essential because well developed ML model could potentially bring a lot of value to the company.
During the modeling stage, Data Scientist is solving optimization task: with a given dataset, targets — maximize selected metric. The process is complicated an, it requires different types of skills:
Feature engineering sometimes treated as art, not a science;Hyperparameters optimization requires a deep understanding of algorithms and core ML concepts;Software engineering skills are needed to make output code easy to understand and deploy.
Feature engineering sometimes treated as art, not a science;
Hyperparameters optimization requires a deep understanding of algorithms and core ML concepts;
Software engineering skills are needed to make output code easy to understand and deploy.
AutoML is aimed to help us here.
The input of AutoML is data and task (classification, regression, recommendations, etc.), output — production-ready model, which is capable of predicting unseen data. Every decision in the data-driven pipeline is a hyperparameter. The very idea of AutoML is to find such hyperparameters, which could give a good score in a reasonable amount of time.
AutoML picks a strategy for preprocessing the data: how to deal with imbalanced data; how to fill missing values; remove, replace or keep outliers; how to encode categories and multicategories columns; how to avoid target leakage; how to prevent memory errors; etc.
AutoML generates new features and selects meaningful ones;
AutoML selects model (Linear models, K-Nearest Neighbors, Gradient Boosting, Neural Nets, etc.);
AutoML tune hyperparameters of model chosen (for example, number of trees and subsampling for tree-based models or architecture, learning rate and number of epochs for Neural Nets);
AutoML makes a stable ensemble of models to increase the score if it’s possible.
More companies nowadays either start collecting data, or they want to realize the potential of collected data: they want to get a value from it. On the other hand, there are no too many Data Scientists with a propper background to fulfill the demand, so the gap arises. AutoML could potentially fill this gap.
But could boxed solutions bring any value to a company? In my opinion, the answer is “No.”
These companies need a process, but AutoML is just a tool. Advanced tools couldn’t fill the absence of strategy. Before starting using AutoML, consider a project with a consulting firm, which could help you develop a Data Science strategy at first. It is not a coincidence that most of the AutoML solution providers are doing consulting. Some sort foot-in-the-door over here.
According to 2018 Kaggle ML and Data Science Survey 15–26% of the time of a typical data science project, time is devoted to model building or model selection. It’s demanding task both in terms of “man-hours” and computational hours. If the objective or data changes (for example, new features will be added), the process should be repeated. AutoML could help Data Scientists within a company to save time and spend it more on more important stuff (like sword fighting on chairs).
However, if the modeling part for the Data Science team is not the most critical task, you have an apparent problem in processes in your company. Commonly, even a small increase in the model’s performance could potentially earn tons of money for your firm, the modeling time in such case is well-spent time:
Oversimplified rules:if (Gain from model > Costs of DS team time) → Time savings are not needed.if (Gain from model <= Costs of DS team time) → Are you solving the right problem? 🤔
It’s a good idea to write scripts for everyday tasks of your DS team and save time in the future instead of using boxed solutions. I’ve written several scripts for automation of routine tasks: automatic feature generation, feature selection, model training, and hyperparameters tuning, which I’m using every day now.
Unfortunately, we don’t have any useful benchmarks “Tabular AutoML vs. Humans”, except “An Open Source AutoML Benchmark”. The authors of papers compared the performance of several AutoML libraries with a performance of tuned Random Forest. It was published several months ago, on the 1st of July 2019.
I was curious and decided to do my benchmarks. I compared my performance with a performance of AutoML solutions on three datasets of binary classification: credit, KDD Upselling, and mortgages. I split dataset in train (random 60% of the data with stratification by target) and test part (remaining 40% of data).
My baseline solution was relatively simple. I didn’t dig into the data, nor create any advanced features:
5-StratifiedKFold;Catboost Encoder for categorical columns (if you aren’t familiar with CatBoost encoder, check my previous post: Benchmarking Categorical Encoders);Mathematical operations (+-*/) for numerical columns pairs. New features limit: 500;Model: LightGBM with defaults parameters;Blending OOF ranked predictions.
5-StratifiedKFold;
Catboost Encoder for categorical columns (if you aren’t familiar with CatBoost encoder, check my previous post: Benchmarking Categorical Encoders);
Mathematical operations (+-*/) for numerical columns pairs. New features limit: 500;
Model: LightGBM with defaults parameters;
Blending OOF ranked predictions.
I used two standard libraries for AutoML: H2O and TPOT. I trained them for several time intervals: starting from 15 minutes up to 6 hours. With the following metric, I got surprising results:
Score = (ROC AUC / ROC AUC of my baseline) * 100%
Firstly, in nearly all cases, my baseline beat AutoML. I was a little bit sad because I’ve planned to chill a little bit in office, while AutoML was doing all the dirty work, but whatever 😒
Secondly, the scores of AutoML weren’t improving over time, which means that it doesn’t matter how long we wait: it will have the same low scores in 15 minutes as in 6 hours.
AutoML is not about high scores.
UPD 16.10.2019: I want to add that my benchmark is not the final point in the “Human vs AutoML” competition. We clearly need more discussion on metric (should it be only scores? scores + time? amount of CO2 emissions during meta-learning, etc) and also more benchmarks on different datasets and tasks.
If your company wants to play with its data, for the first time, consider hiring a consultant.You should automate your work as much as possible...... yet boxed solutions don’t seem like the right choice because of relatively low scores.
If your company wants to play with its data, for the first time, consider hiring a consultant.
You should automate your work as much as possible...
... yet boxed solutions don’t seem like the right choice because of relatively low scores.
In this post, I was talking about tools, yet it’s important to remember that the modeling part is just a part of the whole data science project pipeline. I like the analogy in which project is treated like a car. In such a way, the output of the modeling part — Machine Learning model — is an engine.
Engine, no doubts, is essential, but it is not the whole car. You may spend a lot of time engineering incredible, thoughtful, and sophisticated features, selecting architecture of your Neural Net or tuning parameters of Random Forest and, as a result, create powerful engine. But if you haven’t paid attention to other parts of your car, all your work might be useless.
The model itself could show high score, but it won’t be used because you were solving the wrong problem (business understanding), or data was biased, and you have to retrain it (data exploration) or model is too complex, so it couldn’t be used in production (deployment stage).
In the end, you may find yourself in a silly situation: after days or weeks of hard modeling work, you are driving a slow robust bike with a sports car engine in the basket.
Tools are essential; strategy is vital.