Complete List of Feature Engineering Methods: 40 Techniques, 10 Categories
By: Michelangiolo Mazzeschi
Tags: Encoding, Feature Engineering, Feature Extraction, Embedding, Feature Selection

Why feature engineering?

In reality, feature engineering becomes vital when you start transitioning into Big Data. Big Data analytics is and will always remain in high demand because, as of now, there is almost no way you can learn how to work on a gigantic dataset on your own. The only way to start approaching big data is with a team of experts in the working place. It is the real all-hands-on-deck experience of data science.

Working with small datasets is not computationally intensive. Because you can basically train your model using either your own GPU or the one made available for free from Google or other cloud service providers without any hassle (in Google Colab, for example, you can access it by clicking on a button), there are almost no limitations in the number of times you can train the model. Even for CNN, which requires a lot of computational power, you can make the training in a few hours top, when the dataset is small.

Why Big Data is different from small samples

With BIG DATA is different: every single choice has to be made very carefully, as you do not have many shots at attempting the models. Every single attempt costs a lot of money and time, simply put: there is no room for errors. With feature engineering, you have an idea of how to tune your model or edit your data before any attempt.

For example, if your data has too many outliers, you need to remove them to have a model that generalizes well. Another example is that if there are features which are very similar (their correlation is close to 1) you can delete one of them as it will only require more GPU to train, and will also potentially make the model perform worst.

GPT-3 for example took 4 Million to train (this is, of course, an exaggeration, but you understand what I mean). Imagine the work that the team behind it put in it before spending this much on one attempt. In conclusion, before starting to train your model, you have to be SURE on which tuning and model to use.

This is the most recent list of features I have been able to gather up to now. I will not explain individual features, because the list is massive, my goal is to properly organize every feature in a functioning hierarchy:

Feature Extraction

Notice that I am using the term feature engineering to encompass both feature extraction and feature selection. Feature extraction means to increase (or to find out from nothing) the number of features, while feature selection means to diminish its number.

1. Encoding

With encoding, I am transforming data from categorical to numerical. This becomes very important when we have text instead of numbers, basically the chaos of unstructured data. Techniques like Embeddings Neural Network become very useful. These are the most common techniques, the continuous vectors is the most advanced between them:

One_Hot (ordered)

One_Hot (disordered)

Index-Based encoding

Continuous Vectors

2. Feature Split

By feature split I am transforming a feature into two or more features, increasing the number of features in the model. There is not a rule to perform a split, it depends on the data.

3. Imputation Methods

Imputation is a statistical method used to fill missing values from data. Because while working with data this is very common, there are several ways of filling missing data, depending on the data you are working with. Time-series would likely require Interpolation, while Cross-Sectional data Mean Imputation.

Numerical Imputation

Categorical Imputation

Random Sample Imputation

End of Distribution Imputation

Feature Selection

4. Filter Methods

Filter Methods allow the selection or the deselection of entire features (columns, essentially) based on the passing score of some metrics. For example, if the variance (meaning that the values are too dispersed) of a column is too high, we have good reasons to avoid putting that column in the model, as it will likely diminish its accuracy. The same applies to different metrics.

Variance Threshold

MAD_Mean Absolute Difference

Fisher’s Score

Correlation Coefficient

AM-GM Inequality

Outlier Detection by Standard Deviation

Outlier Detection by Percentile

Chi-Square Test

Information Gain

5. Wrapper Methods

Wrapper methods group several techniques that use an R Squared value to measure whether a feature should be conserved or not. These techniques work by iteratively using features while monitoring a change in score, hence they work recursively:

Exhaustive Feature Selection

Forward Regression

Backward Regression/RFE_Recursive Feature Elimination

Stepwise Regression

Bi-directional elimination

:Numerical Imputation

A list of Numerical Imputation techniques.

Interpolation

Extrapolation

Mean Imputation

Substitution

Hot Deck Imputation

Cold Deck Imputation

Regression Imputation

Stochastic Regression Imputation

6. Binning Methods

Binning methods are one way of diminishing the complexity of data. We can group data together and approximate the value of a group of data by applying a function on it (for example an average):

Numerical Binning

Categorical Binning

7. Grouping Operations Methods

Grouping operations is a way of applying a function to a specific group of values, using an algorithm called group by:

Categorical Grouping

Numerical Grouping

8. Embedded Methods

Embedded methods are ways of performing feature selection while training the model. This is common in neural networks, where, automatically, features are selected or deselected with Normalization techniques:

L1 Normalization

L2 Normalization

Random Forest Importance

9. Hybrid Methods

Hybrid methods are a combination of different techniques (that you can pick from the list above) to perform feature engineering.

Feature Editing

Although this denomination does not exist, I classify under this section all methods which allow editing data, rather than increasing or decreasing the number of features.

10. Feature Scaling

Features scaling is a common feature engineering technique when building regression models: data needs to be placed in the same meter to work properly.