Decision Trees and Random Forests: | by Levi Raichik | Towards Data Science

During my time learning about decision trees and random forests, I have noticed that a lot of the hyper-parameters are widely discussed and used. Max_depth, min_samples_leaf etc., including the hyper-parameters that are only for random forests as well. One hyper-parameter that seems to get much less attention is min_impurity_decrease.
To understand this, you first have to understand what the basic criterion hyper-parameter does. The criterion used measures the impurity of each split to figure out the information gain by doing that split. It can then decide the best split to do. When the impurity is 0, then all the items in that leaf are correctly classified.
A longer explanation on Gini index and Entropy can be seen in this article:
towardsdatascience.com
What min_impurity_decrease does is stop a split if the amount of decrease from the split is less then the entered amount. This will be more clear to understand as we go on to visualize it.
Let us take a look at a decision tree I made, using the iris dataset and all of the default hyper-parameters. Just a note, due to how this dataset is, the scores are almost perfect from this tree for training and testing. Therefore I will not be showing the scores for each model run as in this article I am just trying to show how to use the min_impurity_decrease hyper-parameter.
Here is the visualization of the decision tree for the training data, with the Gini index for each node and leaf:
There are 120 samples in the training set and there are 3 types of irises that we are trying to classify. Value = [0, 41, 39] for the right side on the first split means that from the first group none of the items are being classified wrongly, and from group 2 and 3 we have almost an equal amount. Therefore the impurity of that node is rounded to .5 which means that about half of them are being correctly classified and about half of them are being incorrectly classified.
One of the things we try to minimize in modeling is over-fitting. The reason for this is, if we over-fit on our training data then we can miss-classify on our testing data, or on real data. Therefore we want to minimize leafs that are splitting down to very small amounts.
Here is where min_impurity_decrease can help us as well. If the impurity of a split goes up then that would mean we are making a split which is not helping us and is instead over-fitting to the training data.
Let us look at an example from the above decision tree:
This is the 3rd level of splits being done. As can be seen on the right side, we made a split for 5 miss-classified items. On the left side of that split, there are 4 correct and 4 incorrect, with a Gini index of .5. On the right side, there are 34 correct and 1 incorrect. We seem to be over-fitting here and not even gaining a lot by doing so.On the left side, we have 36 correct and one incorrect, we then make a split to catch that last one. This is also over-fitting to our training data.
How can we handle both of these splits?
How does this hyper-parameter get calculated? If we check the documentation for it, which can be found here, then we would see this:
min_impurity_decrease : float, optional (default=0.)A node will be split if this split induces a decrease of the impurity greater than or equal to this value.The weighted impurity decrease equation is the following::N_t / N * (impurity — N_t_R / N_t * right_impurity — N_t_L / N_t * left_impurity)where ``N`` is the total number of samples, ``N_t`` is the number ofsamples at the current node, ``N_t_L`` is the number of samples in the left child, and ``N_t_R`` is the number of samples in the right child.``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum, if ``sample_weight`` is passed.
Let us break down this calculation so we can better understand it:N_t_R / N_t * right_impurity: Taking the impurity of the right side of the split and weighting it by the percentage of samples in the right split from the node.
N_t_L / N_t * left_impurity: Taking the impurity of the left side of the split and weighting it by the percentage of samples in the left split from the node.
We then subtract both of those from the node impurity.Then we weight that amount by the percentage of samples in the node compared to the total number of samples.
Let’s write this out for the right side of the split:
43/120 * (.206 - 35/43 * .056 - 8/43*.5)
The total we get is 0.024149999999999998 or just round it to .024.
43 is the total number of samples in the node that is splitting120 is the total number of samples.206 is the impurity of the node35 is the total number of samples in the right side of the split.056 is the impurity in the right side of the split8 is the total number of samples in the left side of the split.5 is the impurity in the left side of the split
Calculating this all out by hand is not so easy and I made a function that allows you to put in the above numbers and it will calculate the impurity of a split for you:
def get_impurity_score(n=None, n_t=None, n_t_l=None, n_t_r=None, node_impurity=None, left_impurity=None, right_impurity=None):    '''    n = Total number of samples overall    n_t = Samples in the node you are checking out    n_t_l = Samples in the left split from the node    n_t_r = Samples in the right split from the node    node_impurity = Impurity score of the node    left_impurity = Impurity score of the left split from the node    right_impurity = Impurity score of the right split from the node        Returns impurity decrease total of the 2 splits from the node impurity weighted by the total number of samples    '''    if n == None or n_t == None or n_t_l == None or n_t_r == None or node_impurity == None or left_impurity == None or right_impurity == None:        print('-----error------ missing values. Check inputs')    return(n_t / n * (node_impurity - (n_t_r / n_t * right_impurity) - (n_t_l / n_t * left_impurity)))
The impurity score for the left side split is 0.016341666666666668, or rounded to .016.
If we now set our min_impurity_decrease to .025 then neither split should happen and we would lessen our over-fitting in our model.
Our updated tree:
We did now stop both of those splits from happening to help prevent over-fitting in our Decision Tree.
To use this in Random Forests, I would suggest using GridsearchCV to help with finding the hyper-parameters that are good for the model you are trying to build.
In general, I would suggest using this along with other hyper-parameters in preventing over-fitting.
I hope this has helped you with understanding min_impurity_decrease and how it should be used. Please use the comment section below with any questions or comments you may have.