Essential guide to perform Feature Binning using a Decision Tree Model
By: Satyam Kumar
Tags: Feature Engineering, Data Science, Artificial Intelligence, Education, Machine Learning

Essential guide to perform Feature Binning using a Decision Tree Model

Image by Pete Linforth from Pixabay

Feature Engineering is an essential component of a machine learning model development pipeline. A machine learning model understands only numerical vectors, so a data scientist needs to engineer the features to train a robust machine learning model.

Binning or discretization is used to encode a continuous or numerical variable into a categorical variable. Sometimes numerical or continuous features do not work well with non-linear models. So binning of continuous variables introduces non-linearity in the data and tends to improve the performance of the model.

There are various techniques to perform feature binning including the Unsupervised and Supervised methods of Binning.

(Image by Author), Technique to perform feature binning

Read my previous article, to get an in-depth understanding of feature encoding strategies

In this article, we will discuss one of the best strategies to bin the numerical feature using a decision tree model.

What is Feature Binning?

Feature binning refers to a technique to transform or bucket the numerical or continuous feature into a categorical variable.

Pandas come up with a pd.qcut(x, q) function that buckets the continuous feature into q buckets, with equal percentile differences.

df['Sepallength_quartle'] = pd.qcut(df['SepalLengthCm'], 4)

(Image by Author), Bucketed Result Sample

Or one can also use pd.cut(x, bins, labels) function and pass custom bins and labels for bucketing.

These kinds of bucketing strategies do not involve the target variable while computing the bins. Hence the bins night does not have any correlation with the target variable. The aim of feature binning is to introduce non-linearity in the data which may further improve the performance of the model.

Idea:

The main purpose of feature binning is not met, by performing random, count, or quartile-based bucketing. The idea is to find the best set of buckets or bins using a decision tree model that will involve correlation with the target variable.

Data Source: We will be binning the ‘Sepal Length’ feature from the Iris dataset that is downloaded from Kaggle.

Follow the following steps to decide the optimal ‘Sepal Length’ bins:

Train a best-fit Decision Tree model with feature ‘Sepal Length’ as training data and ‘Species’ as the target variable.

X = df[['SepalLengthCm']]

y = df['Species'] params = {'max_depth':[2,3,4], 'min_samples_split':[2,3,5,10]}

clf_dt = DecisionTreeClassifier()

clf = GridSearchCV(clf_dt, param_grid=params, scoring='accuracy')

clf.fit(X, y) clf_dt = DecisionTreeClassifier(clf.best_params_)

Plot a decision tree using Graphviz.

tree.plot_tree(clf_dt, filled=True, feature_names = list(X.columns), class_names=['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'])

plt.show()

Decide the feature buckets from observing the graph nodes.

(Image by Author), Decision Tree representation using Graphviz

From the above graphical representation of the Decision tree model, we can conclude that Iris-setosa has most of the points having sepal length ≤ 5.45, and a majority of Iris-virginica data points have sepal length > 6.15. The majority of Iris-versicolor data points have a sepal length in the range of 4.86 — 6.15.

Using the above observation we can prepare the buckets to bin the ‘Sepal Length’ continuous feature.

bins = [0, 5.45, 6.15, 10]

labels = ['Less Than 5.45','4.86 - 6.15', 'Above 6.16']

df['SepalLengthBucket'] = pd.cut(df['SepalLengthCm'], bins=bins, labels=labels)

(Image by Author), Correlation between Bucketed continuous feature (‘SepalLengthBucket’) and target variable (‘Species’)

The bucketed Sepal Length feature is well-correlated with the target variable and will train a much robust model better than random, count, or quartile-based bucketing strategies.

Conclusion:

Binning of continuous variables introduces non-linearity in the data and tends to improve the performance of the model. The decision tree rule-based bucketing strategy is a handy technique to decide the best set of feature buckets to pick while performing feature binning.

One must keep in mind not to train the decision tree model having larger depth, as it becomes difficult to interpret the feature buckets.

References:

[1] Scikit-learn documentation: https://scikit-learn.org/stable/modules/tree.html