Decision Tree Machine Learning Algorithm from Scratch in Python
By: Dhiraj K
Tags: Decision Tree, Machine Learning, Artificial Intelligence, Python Programming, Entropy

Yes, you read it right !! A tree can take decision !!

Introduction:

When we are implementing the Decision Tree Machine Learning Algorithm using sklearn, we are calling the sklearn library methods. Hence we are not implementing the algorithm from scratch.

In this article, we will be implementing a Decision Tree algorithm without relying on Python’s easy-to-use sklearn library. The goal of this post is to discuss the fundamental mathematics and statistics behind a Decision Tree algorithm model. I hope this will help you understand at a low level, how Decision Tree works in the background.

A decision tree algorithm, is a machine learning technique, for making predictions. As its name suggests, it behaves like a tree structure. The decision tree is built by, repeatedly splitting, training data, into smaller and smaller samples.

Decision Tree from Scratch in Python

Decision Tree works on, the principle of conditions. The algorithm checks conditions, at a node, and split the data, as per the result, of the conditional statement. Decision Tree algorithm belongs to, the family of, supervised machine learning algorithms. It can be used to, build classification, as well as regression models.

In this article, we are going to discuss, how to implement, a simple Decision Tree, from scratch in Python. This means we are not going to use, any machine learning libraries, like sklearn.

Note that this is one of the posts in the series Machine Learning from Scratch. You may like to read other similar posts like Gradient Descent From Scratch, Linear Regression from Scratch, Logistic Regression from Scratch, Neural Network from Scratch in Python

You may like to watch this article as a video, in more detail, as below:

Implementation:

General Terms:

Let us first discuss a few statistical concepts used in this post.

Entropy: The entropy of a dataset, is a measure the impurity, of the dataset

Entropy can also be thought, as a measure of uncertainty. We should try to minimize, the Entropy. The goal of machine learning models is to reduce uncertainty or entropy, as far as possible.

Information Gain: Information gain, is a measure of, how much information, a feature gives us about the classes. Decision Trees algorithm, will always try, to maximize information gain. Feature, that perfectly partitions the data, should give maximum information. A feature, with the highest Information gain, will be used for split first.

Implementation:

Import Libraries:

We are going to import NumPy and the pandas library.

# Import the required libraries

import pandas as pd

import numpy as np

Load Data:

We will be using pandas to load the CSV data to a pandas data frame.

# Load the data

df = pd.read_csv('data-dt.csv')

df.head()

Data for Decision Tree from Scratch in Python

Define the calculate entropy function: we are defining, the calculate entropy function. This function is taking one parameter, the dataset label. After calculation, it is returning, the entropy of the dataset.

Calculate Entropy in Python for Decision Tree

Define the calculate information gain function: This function, is taking three parameters, namely dataset, feature, and label. Here, we are first calculating, the dataset entropy. Then, we are calculating, the weighted feature entropy. Note that, we are calling, the calculate_entropy function, from inside of this function. At the end of this function, we are able to calculate, feature_information_gain, which is equal to, dataset_entropy minus weighted_feature_entropy

Calculate Information Gain in Python for Decision Tree

Define the create decision tree function: A decision tree, recursively, splits the training set into smaller and smaller subsets Decision trees, are trained by, passing data down, from a root node to leaves. The data is repeatedly split, according to feature variables, so that, child nodes are purer. This helps in reducing entropy. A pure node means, all the samples at that node, have the same label.

Define the create decision tree function in Python Recursively

Note the recursive call to create_decision_tree function, towards the end of this function. This is required, as the tree grows recursively.

Endnotes:

In this article, I built a Decision Tree model from scratch without using the sklearn library. However, if you will compare it with sklearn’s implementation, it will give nearly the same result.

The code is uploaded to Github here.

Happy Coding !!