Optimizing a Random Forest
By: Raheel Hussain
Tags: Random Forest, Decision Tree, Decision Tree Classifier, Python, Data Science

Following article consists of the seven parts:

1- What are Decision Trees

2- The approach behind Decision Trees

3- The limitations of Decision Trees and their solutions

4- What are Random Forests

5- Applications of Random Forest Algorithm

6- Optimizing a Random Forest with Code Example

The term Random Forest has been taken rightfully from the beautiful image shown above, which shows a forest, consisting of many trees, big & small, some with many branches/leaves, and some with less.

The Random Forest is the most popular and widely used supervised learning algorithm around for both classification and regression tasks, and there are valid reasons for that, such as:

It Trains faster

It works well with big Data Sets

It is good at handling missing values

It has higher accuracy

It reduces over-fitting by using Multiple Decision Trees

The concept of Random Forest, in machine learning, revolves around multiple Decision Trees grouped together, and certain techniques applied, to yield the results. But let’s start with the intro to the Decision Trees first.

1. Decision Trees

A typical Decision Tree

The Decision Tree is itself an algorithm, and can be used for both Classification and Regression tasks. In contrast to linear models, trees are able to capture non-linear relationships between features and targets (labels), and they don’t require the features to be on the same scale (i.e. standardization).

We can define a decision tree as, predictive models that use rules to calculate and split data, in order to get the predictions. A decision tree starts from a root node, and may consists of many branches, decision nodes & leaves depending on the nature of the data it is applied for.

Click on Classifiers in Machine Learning to understand about Classification cases, and Predictions using Linear Regression to understand about Regression cases.

2. The Decision Tree Approach

A decision tree uses Top-Down approach as below:

Select the best candidate attribute to be Root Node (say A)

“A” becomes decision attribute for other nodes (the split starts from A)

For each value of A, create a new descendant node (decision node or leaf)

If the data gets perfectly classified (node can’t be further split), then Stop

Else continue iterating the same process

As mentioned above, the first thing a decision tree does, is to select the best candidate attribute to be the root node. That is because, the least number of comparisons it does, the better. So the systematic way will be, to select one best attribute out of many, such that the resulting tree becomes as simple as possible.

To do so, one of the below two methods are employed:

1- Information Gain, that works on top of Entropy

2- Gini-Index or simply Gini

Information Gain

Entropy & Information Gain isn’t preferred, as it makes use of log function, which increases the computational complexity. While Gini is also the default criterion in Random Forest. Though the concept of Entropy is equally important.

In the Information Gain method, we first would have to calculate the Entropy. Once Entropy is calculated, we define our equation for Information Gain for each attribute respectively.



Entropy means, chaos, uncertainty, unpredictability in the data.

Consider the example of a following simple Dataset, where the resulting attribute/label reads, either “A” or “B”

Decision tree on a data set

Note, that in the above image, attribute “X” has been chosen to become the root node, now let’s take “Y” and “Z” to be our root nodes respectively, and see what kind of decision tree comes out of it.

Notice the number of splits are 5 and 6, for Y and Z, selected as root node respectively. Choosing X as the root node, we only had 3 splits. Therefore, we can easily conclude, that X is the best candidate to become the root node.

While this is a fairly simple example, and by visualizing it, some of us can easily detect the best candidate to be the root node.

But what about complex data sets with lots of attributes?

The decision tree uses two equations to come up with the solution, the first one being Entropy defined as:

Equation of Entropy

T he equation reads as, the entropy of a set “S” is defined as the summation of the probability “p” of an outcome “i”, which goes from 1 … n, times the logarithm (of base 2) of that probability “p” of the outcome “i”.

To make this further understandable for the above data set “S”, there are two probabilities for the outcome “i”. The probability of result “A” is 5/8, which is 0.625 and the probability of “B” is 3/8, which is 0.375.

The value of probability will always be between 0 to 1. For example, if the probability of result “A” is 0.0 or 1.0 then the entropy is lowest. While the value of entropy is highest, if the probability is 0.5. That means that there is equal distribution of classes.

The above equation can be further broken down to the sum of probabilities of both positive and negative outcomes of a given attribute, as below:

The graph below, shows the probability when X=1 that is 4/4=1, and X=0 that is 1/4=0.25, as well as for Y and Z.

Entropy v. Probability graph for possible values of X, Y & Z.

Further, we calculate the Information Gain that can be defined as the measurement of the decrease in entropy of an attribute say “X” due to categorization or sorting based on its values. below is the equation:

Where, S is the set of examples in data set, X is the attribute, every value v in all possible values of X which is Values(X). S with subscript v is subset of S for attribute X.

Hence, a Decision Tree makes its decisions, is based on Entropy, which is basically the measure of how unpredictable and random the data is.

Gini Index

The Gini Index or simply Gini is the measure of impurity. In simple words, it is the probability of a particular independent variable wrongly classified when it is randomly chosen.

If Gini is 0.5, then it means that the impurity is at the highest because there is an equal distribution of classes. If Gini is 0, then this is the purest distribution, or in other words, impurity is 0.

The formula of Gini which is as follows:

The equation reads as, 1 minus, summation of all possible values of a feature i, where i ranges from 1 to n, into probability of i squared.

Considering the data-set, mentioned in the first image, let’s apply the above equation for calculating the Gini Index.

In the data-set used, the values “1” for X has been repeated 4/8 times, hence the weighted sum we’ve got 4/8 * 0.625. While Gini Index for Z is same as Y.

Below is an example of how a decision tree goes down using sklearn’s breast cancer dataset along with sample code.

Notice that there are around 30 features in the breast_cancer dataset, while the estimator tree uses only a few. Likewise, each estimators out of the 10 selected estimators in the code, would use different set of randomly selected features.

You can learn more about this in the sklearn’s documentation at the link:

scikit-learn Decision Tree Classifier

Code:

Result: