Decision Tree Fundamentals. Learning about Gini Impurity, Entropy... | by Huy Bui | Towards Data Science

When talking about the decision trees, I always imagine a list of questions I would ask my girlfriend when she does not know what she wants for dinner: Do you want to eat something with the noodle? How much do you want to spend? Asian or Western? Healthy or junk food?
Making a list of questions to narrow down the options is essentially the idea behind decision trees. More formally, the decision tree is the algorithm that partitions the observations into similar data points based on their features.
The decision tree is a supervised learning model that has the tree-like structured, that is, it contains the root, parent/children nodes, and leaves. The decision tree can be used for classification or regression problems. The following diagram below is a short version of the binary classification tree that I used to determine if she wants fast food or not.
There are 2 popular tree building-algorithm out there: Classification and Regression Tree (CART), and ID3. The main difference between these two models is the cost function that they use. The cost function decides which question to ask and how each node being split. The pseudocode for constructing a decision tree is:
Chose a feature that has the optimal index. The index is calculated using the cost function which I will introduce in the next sectionSplit the dataset based on the chosen featureRepeats the process until it reaches to the leaves (Or meet the stopping criteria)
Chose a feature that has the optimal index. The index is calculated using the cost function which I will introduce in the next section
Split the dataset based on the chosen feature
Repeats the process until it reaches to the leaves (Or meet the stopping criteria)
There are 2 cost functions that we will talk about in this post: the Gini Impurity and Entropy
Gini impurity is the lost function being used in the CART method. Gini Impurity measures how much noise a category has. For starters, each data feature may consist of many categories. For example, the weather feature can have categories: rain, sunny, or snowy; a numerical feature such as grade can be divide into 2 blocks: <70 or ≥70. Gini impurity can be calculated by the following formula:
Note that the maximum Gini Impurity is 0.5. This can be check with some knowledge of Calculus. I created a toy dataset to better illustrate the effect of cost function on the decision tree.
import pandas as pdclassification=['Freshman','Freshman','Sophomore','Junior','Freshman','Sophomore']hour_of_practice=['>2h','>2h','>2h','<2h','>2h','<2h']pass_the_quiz=['Yes','Yes','Yes', 'Yes', 'No','No']df=pd.DataFrame({'Classification':classification,                 'hour of practice':hour_of_practice,                 "Pass the quiz":pass_the_quiz })df
Observation: There are 3 freshmen (F), 2 sophomores (S), and 1 junior (J). From the data, the freshmen and sophomore pass the test if they study more than 2h and fail if they study less. The junior cheats the system and always passes the test.
Let’s compute the Gini Impurity of the classification column:
The Gini values tell us the value of noises present in the data set. In this case, the junior has 0 noise since we know all the junior will pass the test. On the other hand, the sophomore has the maximum noise.
Gini Index combines the category noises together to get the feature noise. Gini Index is the weighted sum of Gini Impurity based on the corresponding fraction of the category in the feature. The formula is:
Applying Gini Index to classification we obtain:
You should be able to calculate the Gini Index of the hour of study feature by hand. For the sake of variety, I created the code below to calculate Gini Impurity and Gini Index:
# Input Format(df, feature name, category name ,target name, list of all classes)# where df is the current node observationdef gini(df,feature,category,target,classes_list):    df=df[df[feature]==category]    def P_i_K(i):        return len(df[df[target]==i])/len(df)            result=1-sum([(P_i_K(i))**2 for i in classes_list])    return result# Input Format(df, feature name, target name, list of all classes)def Gini_index(df,feature,target,classes_list):    def P_k_a(category):        return len(df[df[feature]==category])/len(df)        result=0    for category in df[feature].unique():        gini_value=gini(df,feature,category,target,classes_list)         P_k_a_value=P_k_a(category)        result+=gini_value*P_k_a_valuereturn resultprint("Gini Index of Classification",      Gini_index(df,"Classification","Pass the quiz",['Yes','No']))print("Gini Index of hour of practice",       Gini_index(df,"hour of practice","Pass the quiz",['Yes','No']))>>> Gini Index of Classification 0.38888888888888884>>> Gini Index of hour of practice 0.41666666666666663
Since Classification has less noise than the hour of practice, the first split goes for the Classification feature. We will visualize the tree after discussing the entropy concept.
In physics, entropy represents the unpredictability of a random variable. The chance of having Head or Tail from a fair coin is 50/50, and so its entropy value is 1, which is the highest value for randomness. On the other hand, having a value 0 indicates the corresponding event is destined. Entropy is similar to Gini Impurity in the sense that they both indicate the unpredictability of the category. The formula for entropy is:
Applying this formula to freshmen, sophomore, and junior we obtained:
As mention above, sophomore entropy acts like a fair coin and so it has the highest value. Freshmen have a value very closed to 1 since its classes are unbalanced.
After obtaining the entropy for each category, we can combine them to get the information gain values for the features. The more information we gain, the better. The formula is:
Applying this formula to our “Classification” feature yields:
I also provide the code to calculate entropy and the information gain:
# Input Format(df,feature name K,category name,target name, list of all classes)# Pass feature, category None if want to find the entropy of the whole observationimport mathdef entropy(df,feature,category,target,classes_list):        if (feature!=None)|(category!=None):        df=df[df[feature]==category]            def P_i_k(class_):        return len(df[df[target]==class_])/len(df)        result=0    for class_ in classes_list:        P_i_k_value=P_i_k(class_)        if P_i_k_value!=0:            result+=P_i_k_value*math.log(P_i_k_value,2)    return -1*result# Input Format(df,feature name K,category name,target name,list of all classes)def InfoGain(df,feature,target,classes_list):    H_T=entropy(df,None,None,target,classes_list)        def P_i_a(category,feature):        return len(df[df[feature]==category])/len(df)        result=0        for category in df[feature].unique():        result+=P_i_a(category, feature)*entropy(df,feature,category,target,classes_list)    result=H_T-result    return resultprint("Information Gain of Classification",       InfoGain(df,'Classification','Pass the quiz',['Yes','No']))print("Information Gain of hour of practice",       InfoGain(df,'hour of practice','Pass the quiz',['Yes','No']))>>>Information Gain of Classification 0.12581458369391152>>>Information Gain of hour of practice 0.044110417748401076
Thus, Classification has more information and so this will be our first split.
In both methods, Classification is the first pick. The second split is the remaining feature, the hour of study, and we obtain the following tree:
Note that this tree is extremely biased because the data set has only 6 observations. A decision tree on real data is much bigger and more complicated.
As the data getting more complex, the decision tree also expands. Nonetheless, if we keep the tree growing until all the training data is classified, our model will be overfitting. Therefore, learning when to stop is very important. Let’s take a look at some commonly used criteria:
The number of observations in the nodes: The ideal upper bound is 5% of the total training dataset.
The node’s purity: The Gini index shows how much noise each feature has for the current dataset and then choose the minimum noise feature to apply recursion. We can set the maximum bar for the Gini Index here as the stopping criteria to notify the branch that it is time to make the decision.
The tree’s depth: we can pre-specified the limit of the depth so that the tree won't expand excessively when facing complex datasets.
Max feature: Since the tree is split by feature, reducing the number of features will result in reducing the size of the tree. Only choosing the feature that has a high Information Gain or low Gini Index can be a good idea.
As we see how the tree was constructed and how it was tuned, we can draw some conclusion about the decision tree:
It is very easy to explain. The decision tree resembles how humans making decisions. Thus, the decision tree is a simple model that can bring great machine learning transparency to the business.
It does not require scaling/normalizing the data, because there is no calculation that involves using data values.
May not create the best tree. The reason is the construction behind the decision tree only focuses on the optimal feature at each node. There is no guarantee the final result will be an optimal solution.
Maybe overfitting. It is very hard to stop the tree from having sub-samples being stuck at some nodes. Having a tree that learns from a particular group will cause overfitting.
You can visit my GitHub repository to download the code. Thanks for reading!