Introduction to decision tree classifiers from scikit-learn
By: Philip Wilkinson
Tags: Python, Decision Tree, Classification, Scikit Learn, Data Science

There are plenty of articles out there that explain what a decision tree is and what it does:

So here I am going to focus on how a decision tree may be implemented using the scikit-learn library in python on the iris dataset, along with some of the functionality that is useful in analysing the performance of the algorithm.

What is a classifier?

A classifier algorithm is used to map input data to a target variable through decision rules and can be used to predict and understand what characteristics are associated with a specific class or target. This means that it is a supervised machine learning algorithm as we already have the final labels, we just want to know how they may be predicted. For our purpose, we can use the Decision Tree Classifier to predict the type of iris flower we have based on features of: Petal Length, Petal Width, Sepal Length and Sepal Width.

What is a decision tree?

A decision tree a tree like structure whereby an internal node represents an attribute, a branch represents a decision rule, and the leaf nodes represent an outcome. This works by splitting the data into separate partitions according to an attribute selection measure, which in this case is the Gini index (although we can change this to information gain if we wanted). This essentially means that we each split aims toreduce Gini impurity which measures how impure a node is according to incorrectly classified results.

Implementing a decision tree

We first of all want to get the data into the correct format so that we can create our decision tree. Here, we will use the iris dataset from the sklearn datasets databases which is quite simple and works as a showcase for how to implement a decision tree classifier.

The good thing about the Decision Tree Classifier from scikit-learn is that the target variable can be categorical or numerical. For clarity purpose, given the iris dataset, I prefer to keep the categorical nature of the flowers as it is simpler to interpret later on, although the labels can be brought in later if so desired. The following code can therefore be used to import the dataset here:

import pandas as pd

import numpy as np

from sklearn.datasets import load_iris #load in the data

data = load_iris()

#convert to a dataframe

df = pd.DataFrame(data.data, columns = data.feature_names)

#create the species column

df['Species'] = data.target #replace this with the actual names

target = np.unique(data.target)

target_names = np.unique(data.target_names)

targets = dict(zip(target, target_names))

df['Species'] = df['Species'].replace(targets)

The next thing we want to do is to extract our training and test dataset. The purpose of this is to ensure that the model is not trained on all the available data so that we can test how it performs on unseen data. If we were to use all the data as the training data then we may end up overfitting the model, meaning that it may perform poorly on unseen data.

#extract features and target variables

x = df.drop(columns="Species")

y = df["Species"] #save the feature name and target variables

feature_names = x.columns

labels = y.unique() #split the dataset

from sklearn.model_selection import train_test_split

X_train, test_x, y_train, test_lab = train_test_split(x,y,

test_size = 0.4,

random_state = 42)

Now that we have the data in the correct format, we can start to create the decision tree so that we can try to predict the classification of the different flowers. To this end, the first thing to do is to import the DecisionTreeClassifier from the sklearn package. For which, more information can be found here.

from sklearn.tree import DecisionTreeClassifier

The next thing to do is then to apply this to training data. For this purpose, the classifier is assigned to clf and set max_depth = 3 and random_state = 42 . Here, the max_depth parameter is the maximum depth the tree, which we control to ensure there is no overfitting and that we can easily follow how the final result was achieved. The random_state parameter ensures that the results can be replicated in further analyses.

We then fit algorithm to the training data:

clf = DecisionTreeClassifier(max_depth =3, random_state = 42)

clf.fit(X_train, y_train)

We want to be able to understand how the algorithm has behaved, which one of the positives of using a decision tree classifier is that the output is intuitive to understand and can be easily visualised.

This can be done in two ways:

As a tree diagram: