Decision Trees — Machine Learning in Python | Towards Data Science

In this article, we will look into Decision Trees, another basic and important Machine Learning algorithm to have in your ML algorithms toolbox.
We will start by looking into what it does and how it works. We won’t be looking into the mathematics part as it’s another article on its own. We will then implement it using scikit-learn
What are Decision Trees
Python Implementation (Dummy Data)
Python Implementation (Real-World Dataset)
Plotting the Decision Tree
Applications
Conclusion
What you can do
Other ML Algorithms
This is part of my 30-Day Article Writing Challenge. Feel free to check out the articles on my pledge post:
nouman10.medium.com
A Decision Tree is a machine learning algorithm used for classification as well as regression purposes (although, in this article, we will be focusing on classification). As the name suggests, it does behave just like a tree.
It works on the basis of conditions. Every condition breaks the training data into two or more smaller sets of training data.
For instance, if we are building an application that decides whether a person’s application for a loan should be accepted or not. We can start by checking the credit score of the applicant. If it’s below some threshold, we reject the application. Otherwise, we check for other things, all the way down until everything is checked and we accept the application.
An example for that would be as follows:
Let’s start by implementing Decision trees on some dummy data.
We start by importing the tree module from scikit-learn and initializing the dummy data and the classifier. We fit the classifier to the data and predict using some new data. The classifier predicts the new data as 1. Let’s plot using the built-in plot_tree in the tree module
tree.plot_tree(clf)
This plots the following tree:
At the root node, we have 5 samples. It checks for the first value in X and if it's less than or equal to 0.5, it classifies the sample as 0. If it’s not, then it checks if the first value in X is less than or equal to 1.5, in which case it assigns the label 1 to it, otherwise 2.
Note that the decision tree doesn’t include any check on the second value in X. This is not an error as the second value is not needed in this case. If the decision tree is able to make all the classifications without the need for all the features, then it can ignore other features.
Let’s now implement it using the iris dataset. The code is the same as above, except we load the iris dataset using scikit-learn
The accuracy, in this case, is 100% as the decision tree is able to build a tree that classifies all the training data accurately.
We won’t be using the built-in plot_tree function for plotting the tree here. In the next section, we will be using graphviz to plot the decision tree instead.
Let’s plot the tree for the above classifier using graphviz .
graphviz allows us to create a decision tree that is more aesthetically pleasing and is easily understandable by even a non-technical person, who hasn’t heard about decision trees before.
We will use the following code to plot the decision tree:
We call the export_graphviz function from the tree module and give it the feature and class names. We can even save the tree to a file but we aren’t doing it in this case. This is the output from the above code:
See how you can easily interpret the decision tree
One of the applications of decision trees involves evaluating prospective growth opportunities for businesses based on historical data.
Another application of decision trees is in the use of demographic data to find prospective clients.
Lenders also use decision trees to predict the probability of a customer defaulting on a loan, by applying predictive model generation using the client’s past data.
Let’s conclude what we have done in this article:
We started with a general explanation of how Decision Trees work
We then implemented it in Python on some dummy data and later on a dataset
We then plotted the decision trees built using scikit-learn
Finally, we looked into the applications of decision trees.
Try to implement Decision Trees from scratch. For this, you need to understand the maths behind Decision Trees
Compare your implementation to the one in scikit-learn
Test the above code on various other datasets.
Feel free to reach out to me if you have any questions. Do follow me as I plan to cover more Machine Learning algorithms in the future
Linear Regression — Machine Learning Algorithms with Implementation in Python
medium.com
K-Nearest Neighbors — Machine Learning Algorithms with Implementation in Python
nouman10.medium.com
K-Means — Machine Learning Algorithms with Implementation in Python
towardsdatascience.com
If you feel that above content was useful for you, do share it and feel free to support me ->