Discretisation Using Decision Trees
By: Akash Dubey
Tags: Machine Learning, Data Science, Model, Artificial Intelligence, Feature Engineering

1. Introduction

Discretisation is the process of transforming continuous variables into discrete variables by creating a set of contiguous intervals that span the range of variable values.

1.1 Discretisation helps handle outliers and highly skewed variables

Discretisation helps handle outliers by placing these values into the lower or higher intervals together with the remaining inlier values of the distribution. Thus, these outlier observations no longer differ from the rest of the values at the tails of the distribution, as they are now all together in the same interval/bucket. In addition, by creating appropriate bins or intervals, discretisation can help spread the values of a skewed variable across a set of bins with an equal number of observations.

1.2 Discretisation approaches

There are several approaches to transform continuous variables into discrete ones. This process is also known as binning, with each bin being each interval. Discretization methods fall into 2 categories: supervised and unsupervised.

Unsupervised methods do not use any information, other than the variable distribution, to create the contiguous bins in which the values will be placed.

Supervised methods typically use target information in order to create bins or intervals.

We will only talk about supervised discretisation method using decision trees here in this article

But before moving to the next step, let’s load a dataset on which we will perform the discretisation.

Discretisation with decision trees

Discretisation with Decision Trees consists of using a decision tree to identify the optimal splitting points that would determine the bins or contiguous intervals:

Step 1: First it trains a decision tree of limited depth (2, 3 or 4) using the variable we want to discretize to predict the target.

Step 2: The original variable values are then replaced by the probability returned by the tree. The probability is the same for all the observations within a single bin, thus replacing by the probability is equivalent to grouping the observations within the cut-off decided by the decision tree.

Advantages :

The probabilistic predictions returned decision tree are monotonically related to the target.

The new bins show decreased entropy, this is the observations within each bucket/bin are more similar to themselves than to those of other buckets/bins.

The tree finds the bins automatically.

Disadvantages :

It may cause over-fitting

More importantly, some tuning of tree parameters might need to be done to obtain the optimal splits (e.g., depth, the minimum number of samples in one partition, the maximum number of partitions, and a minimum information gain). This it can be time-consuming.

Let ’s see how to perform discretization with decision trees using the Titanic dataset.

Import useful Libraries

IN[1]:

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split

2. Load the dataset

IN[2]:

data = pd.read_csv('titanic.csv',usecols =['Age','Fare','Survived'])

data.head()

3. Separate the data into train and test set

IN[3]:

X_train, X_test, y_train, y_test = train_test_split(data[['Age', 'Fare', 'Survived']],data.Survived , test_size = 0.3)

So, assuming that we do not have missing values in the dataset (or even if we have missing data available in the dataset, we have imputed them ). I am leaving this part because my main goal is to show how discretisation work.

So, Now let’s visualize our data such that we gain some insights out of it and understand the variables

4. Let’s build a classification tree using the age to predict Survived in order to discretise the age variable.

IN[4]:

tree_model = DecisionTreeClassifier(max_depth=2) tree_model.fit(X_train.Age.to_frame(), X_train.Survived) X_train['Age_tree']=tree_model.predict_proba(X_train.Age.to_frame())[:,1] X_train.head(10)

Now that we have a classification model using the age variable to predict the Survived variable.

The newly created variable Age_tree contains the probability of the data point belonging to the corresponding class

5. Checking the number of unique values present in Age_tree variable

IN[5]:

X_train.Age_tree.unique()

Why only four probabilities right?

Above in input four, we have mentioned max_depth = 2. A tree of depth 2, makes 2 splits, therefore generating 4 buckets, that is why we see 4 different probabilities in the output above.

6. Check the relationship between the discretized variable Age_tree and the target Survived .

IN[6]:

fig = plt.figure()

fig = X_train.groupby(['Age_tree'])['Survived'].mean().plot()

fig.set_title('Monotonic relationship between discretised Age and target')

fig.set_ylabel('Survived')

Here, we can see a monotonic relationship between the discretised variable and Age_tree the target variable Survived . That plot suggests that Age_tree seems like a good predictor of the target variable Survived .

7. Checking the number of passengers per probabilistic bucket/bin to under the distribution of the discretized variable.

IN[7]:

X_train.groupby(['Age_tree'])['Survived'].count().plot.bar()

Let's check the Age limits buckets generated by the tree by capturing the minimum and maximum age per each probability bucket to get an idea of the bucket cut-offs.

8. Checking Age limit buckets generated by the tree

IN[7]:

pd.concat( [X_train.groupby(['Age_tree'])['Age'].min(),

X_train.groupby(['Age_tree'])['Age'].max()], axis=1)

Thus, the decision tree generated the buckets : 0–11 , 12–15 , 16–63 and

46–80 , with probabilities of survival of 0.51 , 0.81 , 0.37 and 0.10 respectively.

9. Visualizing the tree.

IN[8]:

with open("tree_model.txt", "w") as f:

f = export_graphviz(tree_model, out_file=f) from IPython.display import Image

from IPython.core.display import HTML

PATH = "tree_visualisation.png"

Image(filename = PATH , width=1000, height=1000)

Tree Visualisation

As we can see from the plot, we obtain 4 bins for max_depth=2 .

As I mentioned earlier, there are a number of parameters that we could optimise to obtain the best bin split using decision trees. Below I will optimise the tree depth for a demonstration. But remember that you could also optimise the remaining parameters of the decision tree. Visit sklearn website to see which other parameters can be optimised.

10. Selecting the optimal depth of the tree

I will build trees of different depths, and will calculate the roc-auc determined for the variable and the target for each tree I will then choose the depth that generates the best roc-auc

IN[9]:

score_ls = [] # here I will store the roc auc

score_std_ls = [] # here I will store the standard deviation of the roc_auc for tree_depth in [1,2,3,4]:

tree_model = DecisionTreeClassifier(max_depth=tree_depth)



scores = cross_val_score(tree_model, X_train.Age.to_frame(),

y_train, cv=3, scoring='roc_auc')



score_ls.append(np.mean(scores))



score_std_ls.append(np.std(scores))



temp = pd.concat([pd.Series([1,2,3,4]), pd.Series(score_ls), pd.Series(score_std_ls)], axis=1) temp.columns = ['depth', 'roc_auc_mean', 'roc_auc_std'] print(temp)

Here, we can easily observe that we obtained the best roc-auc using depths of 1 or 2. I will select depth of 2 to proceed.

11. Transform the Age variable using tree

IN[10]:

tree_model = DecisionTreeClassifier(max_depth=2) tree_model.fit(X_train.Age.to_frame(), X_train.Survived) X_train['Age_tree'] = tree_model.predict_proba(X_train.Age.to_frame())[:,1] X_test['Age_tree'] = tree_model.predict_proba(X_test.Age.to_frame())[:,1]

12. Inspecting the transformed age variable in the train set

IN[11]:

X_train.head()

13. Checking the unique values of each bin in the train set

IN[12]:

X_train.Age_tree.unique()

14. Inspecting the transformed age variable in the test set

IN[13]:

X_test.head()

15. Checking the unique values of each bin in the train set

IN[14]:

X_test.Age_tree.unique()

Now, we have successfully discretize the Age variable into four discrete values that might help our model to make better predictions.