The Hardest Part of Data Science? Data.
By: Maor Shlomo
Tags: Big Data, Feature Engineering, Ds In The Real World, Data Science, Machine Learning

The core challenge for data science

It’s not about the algorithms, it’s about the right data to feed them.

Today, when applying data science to business problems, almost no one is inventing a new algorithm (again, not necessarily true to computer vision/audio and NLP use cases). We’ve been using the same models for more than a decade. Gradient boosted trees (XGBoost) were introduced in the previous century, SVMs in the 1960s, and linear regression…probably around 1000BC.

Data science is no different than any other profession that has emerged from new technology and influenced society. It continues to evolve and open endless doors to complex questions and challenges, each one more exciting than the last. And yet we continue to use the same models. The winners of Kaggle competitions prove this to be true by using 30-year-old models to top the leaderboard. If 30-year-old models are winning competitions, it stands to reason that these models are not where the “X factor” comes from.

The truth is, data science has been so focused on trying to generate business value from the newest algorithm, that we’ve missed the most important part: data science is not, and never has been, about models.

Given that the majority of modeling is commoditized and automated, what is the next challenge in building a robust and accurate prediction system? What actually makes a model successful?

Putting the “data” into data science

We’ve all heard the saying “garbage in, garbage out.” If I try to train a model to predict credit default based on the number of characters in a person’s name, I might be successful at getting the model to work (train and predict) but the model would pretty much be randomly guessing.

The main question we need to be asking to determine if a data science project will succeed or not is and always will be: Do we have the right data to train a model to solve the problem we’re trying to solve?

The “right data” can be broken down into multiple sub-questions:

Do we have the right data attributes so that our model can infer realistic patterns and rules? Do we have enough samples to train the model? Did we engineer (integrate, aggregate, transform) the data in the right way before we fed it into the algorithm?

Speaking of engineering…Stanford Professor Andrew Ng accurately said:

“…applied machine learning is basically feature engineering.”

What did Professor Ng mean by “feature engineering?”

Due to the buzz in the field, some confusion has arisen around the meaning of different terms. I suggest we break down feature engineering into its two parts:

Feature generation : The initial phase where we search for data to generate features from. We aggregate, apply business logic, and bring multiple sources into one matrix of numerical attributes per sample.

: The initial phase where we search for data to generate features from. We aggregate, apply business logic, and bring multiple sources into one matrix of numerical attributes per sample. Feature standardization: Normalization, scaling, imputation, and encoding. We run transformations on the feature set to make it more “appealing” to the model. These are usually model related. For example, while decision trees could work just fine with raw, unscaled numerical features, linear models will usually require the right imputation and scaling (e.g. to unit variance and zero mean).

I would argue that feature standardization is walking a similar path to the one that models went down. The same way models have become commoditized, feature standardization can be commoditized as well. Just like autoML techniques, you can loop through different models with different standardization techniques until you find the best performing combination. Unlike the search space of possible features to generate, the different combinations of standardization techniques and different models are not as large as we think they are.

To make my point, here is my pseudo-algorithm to automate the selection of a model with feature standardization techniques:

Given a list of possible feature standardization methods, like scaling and imputation methods (mean imputation, unit variance scaling, PCA, …)

And given a list of possible models (linear regression, SVM, XGBoost with 100 decision trees, XGBoost with 1000 decision trees…)

Generate an experiment with a random preprocessor and a random model.

After a couple of iterations, choose the best experiment.

Now obviously it is not as simple as I presented it. The parameter space is large and the number of combinations should be limited — but it is not rocket science. There are a bunch of tools that do a great job at this (like Auto-Sklearn or TPOT). This will only become easier as computing power becomes cheaper and more available.

Feature generation, however, is not at the same place.