Decision Trees Classifier
By: Ahmed Imam
Tags: Decision Tree, Classification, Machine Learning, Decision Tree Classifier, ML

Both of Regression Trees and Classification Trees are a part of CART (Classification And Regression Tree) Algorithm . As we mentioned in Regression Trees article, tree is composed of 3-major parts; root-node, decision-node and terminal/leaf-node.

The criteria used here in node splitting differs from that being used in Regression Trees. As before we will run our example and then learn how the model is being trained.

There are three commonly measures are used in the attribute selection Gini impurity measure, is the one used by CART classifier. For more information on these, see Wikipedia.

Data set being used is iris data set

import numpy as np

import pandas as pd

from sklearn.tree import DecisionTreeClassifier

from sklearn.tree import export_graphviz

from six import StringIO

from IPython.display import Image

# pip/conda install pydotplus

import pydotplus

from sklearn import datasets iris = datasets.load_iris() xList = iris.data # Data will be loaded as an array

labels = iris.target

dataset = pd.DataFrame(data=xList,

columns=iris.feature_names)

dataset['target'] = labels targetNames = iris.target_names

print(targetNames)

print(dataset)

Iris Flowers

How a Binary Decision Tree Generates Predictions

When an observation or row is passed to a non-terminal node, the row answers the node’s question. If it answers yes, the row of attributes is passed to the leaf node below and to the left of the current node. If the row answers no, the row of attributes is passed to the leaf node below and to the right of the current node. The process continues recursively until the row arrives at a terminal (that is, leaf) node where a prediction value is assigned to the row. The value assigned by the leaf node is the mean of the outcomes of the all the training observations that wound up in the leaf node.

How to split the nodes

Classification trees split a node into two sub-nodes. Splitting into sub-nodes will increase the homogeneity of resultant sub-nodes. In other words, we can say that the purity of the node increases with respect to the target variable. The decision tree splits the nodes on all available variables and then selects the split which results in most homogeneous/pure sub-nodes.

There are major measures being used to determine which attribute/feature is used for splitting and which value within this attribute we will start with. Some of these measures are:

Gini index (Default in SciKit Learn Classifier).

Entropy.

Information gain.

We will start with Gini index measure and try to understand it

Gini Index

Gini index is an impurity measure used to evaluate splits in the dataset. It is calculated by getting the sum of the squared probabilities of each class (target-class) within a certain attribute/feature then benig subtacted from one.

How it works

As per below data set.

Algorithm starts with detection of most significant feature (We will describe how it’s being done now) in our case it’s X[2] or (petal width (cm)).

Then, try every value in this feature and get the probability of our target value against this value as follow:

Sort feature X[3] (numerical feature) ascendingly and get average_value between first two points. Let’s no.of elements with (value <= average_value) is n1. Get probability of target-value (0) to be as P1. Get Probability of target-value (1) to be as P2. Get Probability of target-value (2) to be as P3. Calculate the impurity at this value with using Gini Index from the below formula as 𝐺𝐼1

7. Re-do above steps at (value > average_value) and new values of P1, P2 and P3.

8. Let’s no.of elements with (value > average_value) is n2 and Gini index is 𝐺𝐼2

9. Get the value of average Gini impurities for this value from the below formula:

10. Do the above steps with all values within our feature (by getting average_value between each consecutive values).

11. Get the value which represents the min. average Gini impurities

12. This value will represent the Split Point within this feature/attribute.

13. Repeat points 1–12 for the other features and get the best average Gini impurities per each.

14. After finishing get the lowest/better average Gini impurities among the whole features and this will be the master root node of the tree.

15. Repeat the same for sub-nodes .

dataset.head(10)

Train our model

irisTree = DecisionTreeClassifier(max_depth=3, random_state=0)

irisTree.fit(xList, labels)

Get our tree visualized

dot_data = StringIO()

export_graphviz(irisTree, out_file=dot_data,

filled=True, rounded=True,

special_characters=True) graph = pydotplus.graph_from_dot_data(dot_data.getvalue())

Image(graph.create_png())

By checking data set with petal width > 0.8, you will find that, the number of observations is “100” as per first splitting point.

df1 = dataset[dataset['petal width (cm)'] > 0.8] # 'petal width (cm)' == X3

df1.describe()

Graph the first 2-splitting points