Gini Index vs Information Entropy | by Andrew Hershy | Towards Data Science

I recently published a project analyzing the performance between logistic regression and random forest. While working on the random forest component, I wanted expand on measures of impurity/information-gain, particularly Gini Index and Entropy. It’s not as well known as some other topics in machine learning, but I think it adds some valuable perspective to those wanting to understand the foundational mechanics of how decision trees work.
Background
Gini Intuition
Entropy Intuition
Visualization using python
Information Gain Comparison
Practical Takeaways
Decision trees recursively split features with regard to their target variable’s purity. The algorithm is designed to find the optimal point of the most predictive feature in order to split 1 dataset into 2. These 2 new datasets’ target variable will be more pure than the original dataset’s.
“Pure” is the key word here, however. What does that word mean, exactly? In a general sense “purity” can be thought of as how homogenized a group is. But homogeneity can mean different things depending on which mathematical backbone your decision tree runs on. The 2 most popular backbones for decision tree’s decisions are Gini Index and Information Entropy.
These 3 examples below should get the point across:
If we have 4 red gumballs and 0 blue gumballs, that group of 4 is 100% pure.
If we have 2 red and 2 blue, that group is 100% impure.
If we have 3 red and 1 blue, that group is either 75% or 81% pure, if we use Gini or Entropy respectively.
Why does this matter? Depending on which impurity measurement is used, tree classification results can vary. This can make small (or sometimes large) impact on your model.
Let’s start with Gini Index, as it’s a bit easier to understand. According to Wikipedia, the goal is to “measure how often a randomly chosen element from the set would be incorrectly labeled”[1].
To visualize this, let’s go back to the gumball examples. If we decided to arbitrarily label all 4 gumballs as red, how often would one of the gumballs be incorrectly labeled?
The impurity measurement is 0 because we would never incorrectly label any of the 4 red gumballs here. If we arbitrarily chose to label all the balls ‘blue’, then our index would still be 0, because we would always incorrectly label the gumballs.
The gini score is always the same no matter what arbitrary class you take the probabilities of because they always add to 0 in the formula above.
A gini score of 0 is the most pure score possible.
The impurity measurement is 0.5 because we would incorrectly label gumballs wrong about half the time. Because this index is used in binary target variables (0,1), a gini index of 0.5 is the least pure score possible. Half is one type and half is the other. Dividing gini scores by 0.5 can help intuitively understand what the score represents. 0.5/0.5 = 1, meaning the grouping is as impure as possible (in a group with just 2 outcomes).
The impurity measurement here is 0.375. If we divide this by 0.5 for more intuitive understanding we will get 0.75, which is the probability of incorrectly/correctly labeling.
Entropy is more computationally heavy due to the log in the equation. Like gini, The basic idea is to gauge the disorder of a grouping by the target variable. Instead of utilizing simple probabilities, this method takes the log base2 of the probabilities (you can use any log base, however, as long as you’re consistent). The entropy equation uses logarithms because of many advantageous properties. The Main advantage is the additive property it provides. These MIT lecture notes will help grasp the concept more clearly (pg8) [2].
Let’s visualize how entropy works with the same gumball scenarios:
Unsurprisingly, the impurity measurement is 0 for entropy as well. This is the max purity score using information entropy.
The impurity measurement is 1 here, as it’s the maximum impurity obtainable.
The purity/impurity measurement is 0.811 here, a bit worse than the gini score.
Let’s visualize both the Gini and Entropy curves with some code in python:
Below we are making a function to automate gini calculations.
#Gini Function#a and b are the quantities of each classdef gini(a,b):    a1 = (a/(a+b))**2    b1 = (b/(a+b))**2    return 1 - (a1 + b1)
Keeping consistent with our gumball theme, let’s make a loop that calculates the gini score of any conceivable combination of red and blue gumball floats, adding to 4. We will run 10,000 iterations of the gini function above so we can graph the gini curve later.
#Blank listsgini_list = []blue_list = []red_list = []blue_prob_list = []#Looping Gini function on random blue and red float amountsfor x in range (10000): blue = random.uniform(0, 4) red = abs(4-blue) a = gini(red,blue) b = blue/(blue+red) gini_list.append(a) blue_list.append(blue) red_list.append(red) blue_prob_list.append(b)#Dataframe of amount of blue, red, Probability of blue, and gini scoredf = pd.DataFrame({“Blue”: blue_list, “Red”: red_list,”Gini Score”: gini_list, “Probability of Blue”: blue_prob_list})df = df[[‘Red’, ‘Blue’, ‘Probability of Blue’, ‘Gini Score’]]df
The beginning of the dataframe is below. The other 9,994 rows couldn’t fit.
Now we will plot our curve:
plt.scatter(blue_prob_list,gini_list)plt.xlabel(‘Probability of Blue Gumball %’)plt.ylabel(‘Gini’)plt.title(‘Gini Curve’)
#Gini Function#a and b are the quantities of each class. Base is the log base input.def entropy(base,a,b):    try:        var =  abs(((a)/(a+b)) * log(((a)/(a+b)),base)) - (((b)/(a+b)) * log(((b)/(a+b)),base))        return var    except (ValueError):        return 0#Blank listsent_list = []blue_list = []red_list = []blue_prob_list = []#Loop with log base 2for x in range (10000):    blue = random.uniform(0, 4)    red = abs(4-blue)    a = entropy(2,red,blue)    b = blue/(blue+red)    ent_list.append(a)    blue_list.append(blue)    red_list.append(red)    blue_prob_list.append(b)df = pd.DataFrame({"Blue": blue_list, "Red": red_list,"Entropy": ent_list, "Probability of Blue": blue_prob_list})df = df[['Red', 'Blue', 'Probability of Blue', 'Entropy']]df
Gini is blue, Entropy is orange. You will see how these differences are exemplified in information gain in the next section!
Information gain is why impurity is so important. Once we derive the impurity of the dataset, we can see how much information is gained as we go down the tree and measure the impurity of the nodes.
In the example below, we are splitting gumball preference by a particular attribute (let’s say size of the gumball). It gives us the parent/child node relationship:
#Defining Gini info gain function:def gini_info_gain(pa,pb,c1a,c1b,c2a,c2b):    return (gini(pa,pb))-((((c1a+c1b)/(pa+pb))*gini(c1a,c1b)) + (((c2a+c2b)/(pa+pb))*gini(c2a,c2b)))#Running Functiongini_info_gain(22,13,18,2,4,11)
= 0.196 Gini Information gain
0.196/0.467 = 41.97% Gain
#Defining Entropy info gain function:def entropy_info_gain(base,pa,pb,c1a,c1b,c2a,c2b):    return (entropy(base,pa,pb))-((((c1a+c1b)/(pa+pb))*entropy(base,c1a,c1b)) + (((c2a+c2b)/(pa+pb))*entropy(base,c2a,c2b)))#Running Functionentropy_info_gain(2,22,13,18,2,4,11)
= 0.325 Entropy Information gain
0.325/0.952 = 34.14% Gain
Gini has a higher information gain measurement, for this example.
Gini’s maximum impurity is 0.5 and maximum purity is 0Entropy’s maximum impurity is 1 and maximum purity is 0Different decision tree algorithms utilize different impurity metrics: CART uses Gini; ID3 and C4.5 use Entropy. This is worth looking into before you use decision trees /random forests in your model.
Gini’s maximum impurity is 0.5 and maximum purity is 0
Entropy’s maximum impurity is 1 and maximum purity is 0
Different decision tree algorithms utilize different impurity metrics: CART uses Gini; ID3 and C4.5 use Entropy. This is worth looking into before you use decision trees /random forests in your model.
*All the code in this publication can be found on my github here
[1] https://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity
[2] http://web.mit.edu/6.02/www/f2011/handouts/2.pdf
[3] Provost, F., & Fawcett, T. (2013). Data Science for Business: What you need to know about data mining and data-analytic thinking. Köln, CA: O`Reilly.
Please subscribe if you found this helpful. If you enjoy my content, please check out a few other projects:
Is Random Forest better than Logistic Regression? (a comparison)
Excel vs SQL: A Conceptual Comparison
Predicting Cancer with Logistic Regression in Python
Optimize your Investments using Math and Python
Calculating R-squared from scratch (using python)
Word Clouds in Python: Comprehensive Example