Feature Engineering — deep dive into Encoding and Binning techniques | by Satyam Kumar | Towards Data Science

Feature engineering is the most important aspect of a data science model development. There are several categories of features in a raw dataset. Features can be text, date/time, categorical, and continuous variables. For a machine learning model, the dataset needs to be processed in the form of numerical vectors to train it using an ML algorithm.
The objective of this article is to demonstrate feature engineering techniques to transform a categorical feature into a continuous feature and vice-versa.
Feature Binning: Conversion of a continuous variable to categorical.
Feature Encoding: Conversion of a categorical variable to numerical features.
Binning or discretization is used for the transformation of a continuous or numerical variable into a categorical feature. Binning of continuous variable introduces non-linearity and tends to improve the performance of the model. It can be also used to identify missing values or outliers.
There are two types of binning:
Unsupervised Binning: Equal width binning, Equal frequency binning
Supervised Binning: Entropy-based binning
Unsupervised binning is a category of binning that transforms a numerical or continuous variable into categorical bins without considering the target class label into account. Unsupervised binning are of two categories:
This algorithm divides the continuous variable into several categories having bins or range of the same width.
Notations,x = number of categoriesw = width of a categorymax, min = Maximum and Minimun of the list
This algorithm divides the data into various categories having approximately the same number of values. The values of data are distributed equally into the formed categories.
Notations,x = number of categoriesfreq = frequency of a categoryn = number of values in data
Supervised binning is a type of binning that transforms a numerical or continuous variable into a categorical variable considering the target class label into account. It refers to the target class label when selecting discretization cut points. Entropy-based binning is a type of supervised binning.
The entropy-based binning algorithm categorizes the continuous or numerical variable majority of values in a bin or category belong to the same class label. It calculates entropy for target class labels, and it categorizes the split based on maximum information gain.
github.com
Feature Encoding is used for the transformation of a categorical feature into a numerical variable. Most of the ML algorithms cannot handle categorical variables and hence it is important to do feature encoding. There are many encoding techniques used for feature engineering:
Label encoding is an encoding technique to transform categorical variables into numerical variables by assigning a numerical value to each of the categories. Label encoding can be used for Ordinal variables.
Ordinal encoding is an encoding technique to transform an original categorical variable to a numerical variable by ensuring the ordinal nature of the variables is sustained.
Frequency encoding is an encoding technique to transform an original categorical variable to a numerical variable by considering the frequency distribution of the data. It can be useful for nominal features.
Binary encoding is an encoding technique to transform an original categorical variable to a numerical variable by encoding the categories as Integer and then converted into binary code. This method is preferable for variables having a large number of categories.
For a 100 category variable, Label Encoding creating 100 labels each corresponding to a category, but instead binary encoding creating only 7 categories.
One hot encoding technique splits the category each to a column. It creates k different columns each for a category and replaces one column with 1 rest of the columns is 0.
Mean encoding is one of the best techniques to transform categorical variables into numerical variables as it takes the target class label into account. The basic idea is to replace the categorical variable with the mean of its corresponding target variable.
Here the categorical variable that needs to be encoded is the independent variable (IV) and the target class label is the dependent variable (DV).
Steps for mean encoding:
Select a category
Group by the category and obtain aggregated sum (= a)
Group by the category and obtain aggregated total count (= b)
Numerical value for that category = a/b
Feature engineering is a cycle process, nobody can conclude that this feature engineering technique is the best. There is no thumb rule to pick a particular feature engineering — encoding or binning technique during data processing. So one needs to try focusing on feature engineering as per business requirement, try every process several times and pick the best out of it.
Thank You for Reading