Gini Impurity Measure— An intuitive explanation using python | Towards Data Science

The Gini impurity measure is one of the methods used in decision tree algorithms to decide the optimal split from a root node, and subsequent splits.
(Before moving forward you may want to review Making Decisions with Trees)
To put it into context, a decision tree is trying to create sequential questions such that it partitions the data into smaller groups. Once the partition is complete a predictive decision is made at this terminal node (based on a frequency).
Suppose we have a list of observations, that indicates if a person decided to stay home from work. We also have two features, namely if they are sick and their temperature.
We need to choose which feature, emotion or temperature, to split the data on. A Gini Impurity measure will help us make this decision.
Def: Gini Impurity tells us what is the probability of misclassifying an observation.
Note that the lower the Gini the better the split. In other words the lower the likelihood of misclassification.
The chart shown below is a two class problem. We will generalize it later to account for more than 2 groups.
Let Ginx represent the gini index.
Where p1, p2 are class 1 , 2 probabilities, respectively.
Note: p1 + p2 =1
This is not complete yet. The equation above will give us the gini impurity measure for a sub split, but we would like to know the gini impurity measure for the entire split (because the data will be split to the left and right). Therefore, we will need to weigh them accordingly.
Where P_L is the proportion of the split that goes to the left side and p_L1 is analogous to p1 for the left split(same for P_R).
def Ginx(P1,P2):
    #P1 and P2 are the counts for each class after the split
    denom = P1 + P2
    Ginx = 2 * (P1/denom) * (P2/denom)
    return(Ginx)

def Wght_Ginx(G1,G2,PL,PR):
    # G1 G2 are the gini impurity for each split, and PL PR are the proportion of the split
    WG = PL * G1 + PR * G2
    return(WG)

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
X=np.arange(0.0,1.0,0.01)
Y=X[::-1]

Gini=Ginx(X,Y)
plt.plot(X,Gini)
plt.axhline(y=0.5,color='r',linestyle='--')
plt.title('Gini Impurity Graph')
plt.xlabel('P1=1')
plt.ylabel('Impurity Measure')
plt.ylim([0,1.1])
plt.show()

We have a lower bound of zero and an upper bound of one half. The lower the impurity measure the better the split is.
If you look at the graph you will notice that a lower impurity measure is better. When the probability of the observation being class 1 is zero (all the way to the left of the graph) then that means it will always be class 2, and the impurity measure is zero. The same thing occurs on the other end when the probability of the observation being class 1 is 100%.
Let’s go back to our example above and split on both features.
Sick Gini impurity = 2 * (2/3) * (1/3) = 0.444
NotSick Gini Impurity = 2 * (3/5) * (2/5) = 0.48
Weighted Gini Split = (3/8) * SickGini + (5/8) NotSickGini = 0.4665
We are going to hard code the threshold of temperature as Temp ≥ 100.
Temp over impurity = 2 * (3/4) * (1/4) = 0.375
Temp under Impurity = 2 * (3/4) * (1/4) = 0.375
Weighted Gini Split = (4/8) * TempOverGini + (4/8) * TempUnderGini = 0.375
We can see that Temperature has a lower Gini Measure. Therefore we would choose to split on Temperature, because it has the lowest probability of misclassifying the observation.
# Create Sample 
Emotion=['sick','sick','notsick','notsick','notsick','sick','notsick','notsick']
Temperature = ['under','over','under','under','over','over','under','over']
StayHome=['N','Y','Y','N','Y','N','N','Y']
df=pd.DataFrame (list(zip(Emotion,Temperature,StayHome)),
                columns=["Emotion","Temperature","StayHome"])
#sort it by Emotion
df.sort_values(['Emotion'],inplace=True)
print(df)

   Emotion Temperature StayHome
2  notsick       under        Y
3  notsick       under        N
4  notsick        over        Y
6  notsick       under        N
7  notsick        over        Y
0     sick       under        N
1     sick        over        Y
5     sick        over        N

We will verify that our function does return the values we have above.
a=Ginx(2,1)
print("Sick Gini Impurity:",a)
b=Ginx(3,2)
print("Not Sick gini impurity:",b)

Sick Gini Impurity: 0.4444444444444444
Not Sick gini impurity: 0.48

We have concluded the introduction to the gini impurity measure. I hope this brief explanation has given you an insight as to the way a decision tree is making decisions to split the data.
Keep in mind that this is not the only method used, it will depend on the package you use.
Thank you for reading!
Generalization of Gini Impurity
Suppose we have (n) different classes. Then the Gini formula will be :