Decision Trees in Machine Learning | by K G Prajwal | Towards Data Science

Ever wondered about how a human brain works while making a decision? The endless possibilities to be comprehended before going down any road, all the mental calculations it performs regarding the rewards and the penalties that must be considered for achieving the best possible route to a decision. Yes, our brain, truly is a masterpiece and we’ll discuss how we can teach our computer the art of decision making.
Decision Trees are a predictive tool in supervised learning for both classification and regression tasks. They are nowadays called as CART which stands for ‘Classification And Regression Trees’. The decision tree approach splits the dataset based on certain conditions at every step following an algorithm which is to traverse a tree-like graph from root to leaf. At every stage, the nodes of the tree represent the possible test cases for the problem and following along any edge of a node represents a possible solution.
Let’s see what a decision tree looks like.
The diagram illustrates a simple scenario of decision making wherein a person has to decide if he/she is eligible to cast a vote. So, the decision tree starts at the root trying to see the possibilities of making a decision based on the user’s input. Here, it’s a classification problem where a person becomes eligible to vote only if he/she is a citizen of the nation and is above 18 years of age.
You might be thinking right now that this can be implemented using a simple if-else statement in any programming language. But, I assure you it is much more complicated than that.
When it comes to huge and complex datasets there are a variety of parameters to be considered while making decisions throughout the tree. Hence, decision trees come to our rescue to aid us in making these decisions for us.
Let’s see a practical example where we will predict if a person can be hired or not based on their attributes.
Let’s start by importing the dataset.
import numpy as npimport pandas as pdfrom sklearn import tree # import decision tree from scikit learn librarydf = pd.read_csv("PastHires.csv" , header = 0) # Read the dataset
Let’s have a look at what we have here.
To be able to work with machine learning, every alphabetical entry in the dataset has to be converted into numbers.
Convert the ‘Y’ entries in ‘Employed?’, ‘Top-tier school’, ‘Interned’ and ‘Hired’ column to 1 and all ‘N’ entries to 0.
# Convert the alphabetical entries to numerical entriesd = {'Y': 1, 'N': 0}df['Employed?'] = df['Employed?'].map(d)df['Top-tier school'] = df['Top-tier school'].map(d)df['Interned'] = df['Interned'].map(d)df['Hired'] = df['Hired'].map(d)
Let’s also map their qualifications into number. 0 for ‘BS’, 1 for ‘MS’ and 2 for ‘PhD’.
d = {'BS': 0, 'MS': 1, 'PhD': 2}df['Level of Education'] = df['Level of Education'].map(d)
The dataset will now look like,
df.head()
The target column here in the ‘Hired’ column which is the column that we need to predict and the rest of the columns are the features that we consider while making our decision.
So, separate the features column from the target.
# List the factors involved in making the decision to hire or notfeatures = list(df.columns[:6])features
Lets call our features in x and target in y before we build our tree.
y = df["Hired"]X = df[features]
It’s time to build our tree. We will use the DecisionTreeClassifier() method in the tree package to make our tree.
clf = tree.DecisionTreeClassifier() # Decision tree Classifierclf = clf.fit(X,y)
The decision tree is now ready. To visualize the tree we’ll have to install the pydotplus and graphviz package.
pip install pydotplus graphviz
Now, visualize the decision tree.
from IPython.display import Image  from sklearn.externals.six import StringIO  import pydotplusdot_data = StringIO()  tree.export_graphviz(clf, out_file=dot_data,                           feature_names=features)  graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  Image(graph.create_png())
😲Yes! we just obtained an amazing decision tree which may look like this.
Let’s analyze the decision tree we just obtained which is crucial in understanding how this algorithm works. At any point going down the right side of the tree depicts that a person is hired and going down on the left, depicts that the person is not. An observation can be made that a person who has ‘Employed?’ column with a value >0.5 is instantly hired. A person whose ‘Employed?’ column has a value of less than 0.5 but has a value greater than 0.5 in the ‘Interned’ column is hired. Similarly, other nodes in the tree can be evaluated. The tree has certain metrics which it has set for itself to make decisions gradually while traversing down the tree.
Random Forest is nothing but a multitude of decision trees. It takes the number of trees as input and generates that many combinations of decision trees. Finally, the prediction is based on the mode in case of classification or the mean in case of regression.
We’ll use the RandomForestClassifier() method from sci-kit learn library. Let’s generate 10 such decision trees.
from sklearn.ensemble import RandomForestClassifierclf = RandomForestClassifier(n_estimators=10) # Set number of decision treesclf = clf.fit(X, y)
We’ll use this random forest to predict if a person with certain attributes is going to be hired or not. Prediction for a person with the following features:
15 years of experience
has been employed
4 previous employers
PhD as qualification
from a top-tier school, and
has interned
print (clf.predict([[15, 1, 4, 2, 1, 1]]))[1]
Now, let’s predict for another person with features:
1 year of experience
not employed
0 previous employers
BS as qualification
not from a top-tier school
has not interned
print (clf.predict([[1, 0, 0, 0, 0, 0]]))[0]
We see that in the first case, the person who was highly qualified with his/her features for the job didn’t even have to go all the way down the decision tree and was immediately classified as hired. In the second case, The person didn’t meet the features that were required for the company to be hired.
On running predictions for multiple test cases, we can observe that sometimes the value that is predicted changes. This is because the random forest classifier method considers randomly generated trees each time it runs. This is completely normal as we are just using 14 entries in our made-up dataset. With an increase in the size of the dataset, there are increasingly different combinations to be considered which can enhance the purity of the decision tree.
Yay! 🙌 We just built a decision tree that can predict if a person can be hired based on their attributes.
Hope you all understood what and how significant decision trees are in the field of machine learning. For any queries, feel free to contact me on my LinkedIn. The entire code is available on my GitHub.