Deep Dive Into Desision Trees and Random Forest | by Vardaan Bajaj | Towards Data Science

In this post, we’ll go through:
Terminology of Decision TreesWays of Measuring ImpurityCART AlgorithmConstruction of a Decision Tree by hand using CARTWhy choose Random Forest over Decision TreesDiversification of Decision Trees in Random ForestImproving Titanic Dataset classifier using Random Forest
Terminology of Decision Trees
Ways of Measuring Impurity
CART Algorithm
Construction of a Decision Tree by hand using CART
Why choose Random Forest over Decision Trees
Diversification of Decision Trees in Random Forest
Improving Titanic Dataset classifier using Random Forest
In the previous post, we went through Support Vector Machines in great detail and also solved fraudulent credit card transaction dataset from Kaggle. In this post, we’ll be looking at 2 more supervised learning algorithms: Decision Trees and Random Forest. After completing this post, we would have pretty much covered all the widely used supervised machine learning algorithms in the industry.
The term decision tree is pretty much self-explanatory and its working is similar to a human’s decision making power. How do humans make decisions? First of all, humans define the objective. In machine learning, the objective of a particular task is determined by the dataset. Once humans have a defined objective, they answer various questions/perform various tasks in order to achieve that objective. The first task that is performed to achieve the objective is analogous to the root node of the decision tree. It is representative of the objective of the decision tree. The root node branches out to a certain set of options and each of these options have their own set of options. This process perpetuates until we arrive at the final decision. Now, each of these options is analogous to a decision node and the final decision is represented by a leaf node. In almost all the cases, the root node also acts as a decision node. For huge decision trees, the output of a decision node is a number of decision nodes, except at the deepest level of the tree, where it is the leaf node.
Decision Tree modelling is a supervised learning algorithm which can be used for both, continuous and discrete valued datasets, both in regression and classification problems. Most commonly, a decision tree is used as a classifier that recursively partitions data into categories. Decision tree is a directed tree i.e. once we reach particular node in the tree, we cannot backtrack to the previous node i.e. parent nodes are not accessible from child nodes.
After seeing the above example of a decision tree, we can see that the algorithm decides that the best question to ask first is the color of the car. All other questions follow some order as well. The order of asking questions is extremely important otherwise the decision tree can become very complex for a simple scenario like the one shown above. How does the decision tree determine the order of questions it needs to ask? The order of questions being asked depends on which question leads to a good split of the data at that level. The split of data at each node happens in a way similar to binary search. At the root node, we have the entire dataset. The root node should split the data into fed to it into 2 or more groups, where the data in each group has similar attributes. Suppose the root node splits data into 3 groups, then these 3 groups will be the 3 children of the root node and each of these 3 nodes will perform this operation of splitting the data, creating more children and this process is carried on for each node until that node has no more splits to make (leaf node), which is the point where predictions are made.
How does a decision tree decide which question leads to a good split and which question doesn’t? This is quantified by measuring impurity.
The problem of learning an optimal decision tree is NP-Complete, so in order to better mimic the optimal solution, the concept of impurity came into existence. Impurity is a measure of homogeneity of data. Data is said to be pure or homogenous if it only contains a single class. The more the classes in the data, the more impure it is. Each node in the decision tree, except the leaf nodes, contains data which has a potential of splitting into further groups i.e. each node has some sort of impurity related to data. Less impure nodes require less information to describe them and more impure nodes require more information to describe them. Hence, sub-nodes of a node have more purity than their parent nodes.
There are various methods to measure impurity, but the most commonly used are Information Gain and Gini Index. Let’s look at them one by one.
Information Gain is used to determine which feature/attribute gives us the maximum information about a class. It is based on the concept of entropy. Entropy in statistics is analogous to entropy in thermodynamics where it signifies disorder. Higher the entropy, more the randomness (i.e. less purity) and it becomes difficult to draw conclusions from the given information. For totally pure data sample i.e. when only one class exists, the entropy is the least (0) and if data is distributed between all the classes equally, then the entropy is the highest (1).
The binary cross-entropy loss function that we defined for Logistic Regression is quite similar to entropy. For a dataset with ‘c’ different classes, entropy is measured as:
where pi represents the fraction of examples in a given class i. In the example in the image above, there are 2 classes so c=2. p(yes) = 9/14 (0.64) and p(no) = 5/14 (0.36). Once entropy for a node is computed, then the information gain of a specific output given a particular feature is calculated as:
where S is the output class, A is a particular feature of the dataset and P has the same meaning defined above (for pi). Due to the high computational time of the log function, Gini Index is preferred over information gain for practical purposes.
Formally, Gini Index measures the probability of a particular variable being wrongly classified when it is randomly chosen. The probability value of 0 means that the variable couldn’t be wrongly classified and it’s possible only when we have only one output class i.e. the data is 100% pure. As the Gini Index value increases, the chances of misclassification of a particular variable increases since the impurity increases. The information conveyed by these values resonate with the entropy values. For a dataset with ‘c’ classes, Gini Index is defined as:
where pi represents the fraction of examples in a given class i, which is similar to the entropy definition.
There are many more measures of impurity like chi-square, classification error, etc. but the main idea here was to familiarize the readers with impurity measurement methods. One important thing to note here is that these impurity measurement functions play a role analogous to the cost function.
Constructing a decision tree is actually partitioning of the input data features, which leads to 2 or more sub-nodes and this process is carried on recursively for each sub-node. Once this tree is created, features have been partitioned throughout the nodes.
Various decision tree algorithms for classification are compared along the following lines:
(i) The splitting criteria
(ii) Techniques to eliminate/reduce overfitting
(iii) Handling incomplete data
Some of the various algorithms for decision tree construction are CART, ID3, C4.5, C5.0, CHAID, MARS, etc. In this post, we’ll be discussing CART in great detail. CART is one of the most widely used and a highly effective decision tree algorithm.
As the name suggests, CART algorithm is used to generate both, classification and regression decision trees. We’ll be focussing on the classification part here. It is used to solve multi-class classification problem (for binary classification, it generates a binary tree) and uses Gini index as a metric to evaluate the split of a feature node in the decision tree.
In CART algorithm, the objective is to minimize the cost function (Gini Index) at each node. The selection of the input variables/features that decides the specific split for each node is selected in a greedy way to minimize the cost function. In this greedy way, a number of split points with different set of variables/features are considered and that split is chosen which results in the minimum value for the Gini Index (i.e. more homogenous splits) at that node. This process is carried out recursively for all sub-nodes in the tree.
This process can continue on forever and can lead to the formation of many unnecessary sub-nodes, so it needs to be stopped. For this, we count the total number of training examples that pass through each node of the tree. This number is a hyperparameter and is tuned according to the dataset and the optimal choice of this number leads to formation of robust decision trees. To give some intuition, if a node of the tree only has one training example that passes through it, this means that the generated decision tree is suffering from overfitting since it has given so much importance to just one example that it called for its separate node. Such a decision tree needs to reduce its complexity which is achieved using pruning.
Decision tree pruning is a technique to reduce the size of a decision tree by removing sections of the tree that provide little power to the classification objective. 2 popular pruning methods are:
(i) Reduced Error Pruning (bottom-up)
(ii) Cost complexity pruning (top-down)
More about these pruning methods can be found here. Now that we know the underlying logic behind the CART algorithm, let’s see through an example of binary classification, how a decision tree is constructed using CART. Let’s use 10 records of UCI machine learning Zoo Animal Classification dataset to build a decision tree by hand. Given the features “toothed”, “hair”, “breathes”, “legs”, the decision tree should output the species of the animal (mammal/reptile).
Let’s start with each feature one by one. We observe in the data that all the input features are represented by Boolean values.
(i) toothed: Let’s summarize the output (species) wrt the ‘toothed’ feature values.
Gini Index (toothed = true) = 1 — {(5/8)2 + (3/8)2} = 1–0.39–0.14 = 0.47
Gini Index (toothed = false) = 1 — {(1/2)2 + (1/2)2} = 1–0.25 -0.25 = 0.5
To obtain the final Gini index of ‘species’ wrt ‘toothed’, we use the weighted sum of the above calculated values as:
Gini Index(toothed) = (8/10) * 0.47 + (2/10) * 0.5 = 0.38 + 0.1 = 0.48
(ii) hair: Let’s summarize the output (species) wrt the ‘hair’ feature values.
Gini Index(hair = true) = 1 — {(5/5)2 + (0/5)2} = 1–1–0 = 0
Gini Index(hair = false) = 1 — {(1/5)2 + (4/5)2} = 1–0.04–0.64 = 0.32
To obtain the final Gini index of ‘species’ wrt ‘hair’, we use the weighted sum of the above calculated values as:
Gini Index(hair) = (5/10) * 0 + (5/10) * 0.32 = 0.16
(iii) breathes: Let’s summarize the output (species) wrt the ‘breathes’ feature values.
Gini Index (breathes = true) = 1 — {(6/9)2 + (3/9)2} = 1–0.45–0.11 = 0.44
Gini Index (breathes = false) = 1 — {(0/1)2 + (1/1)2} = 1–0–1 = 0
To obtain the final Gini index of ‘species’ wrt ‘breathes’, we use the weighted sum of the above calculated values as:
Gini Index(breathes) = (9/10) * 0.44 + (1/10) * 0 = 0.40
(iv) legs: Let’s summarize the output (species) wrt the ‘legs’ feature values.
Gini Index (legs = true) = 1 — {(6/7)2 + (1/7)2} = 1–0.73–0.02 = 0.25
Gini Index (breathes = false) = 1 — {(0/3)2 + (3/3)2} = 1–0–1 = 0
To obtain the final Gini index of ‘species’ wrt ‘legs’, we use the weighted sum of the above calculated values as:
Gini Index(breathes) = (7/10) * 0.25 + (3/10) * 0 = 0.18
Now that we have calculated Gini Index for the output variable against all 4 input features, we need to choose the root node. The root node is chosen such that it has minimum Gini Index. The lower the Gini Index, the higher the chances of predicting the answer early (due to more purity of the node). So here, we choose hair as the root node. The decision tree at this point looks like:
Let’s first consider the case when ‘hair’ takes the values of TRUE.
Let’s calculate Gini Index for the output against the input features excluding the hair feature, when hair takes the value of TRUE.
(i) toothed: Let’s summarize the output (species) wrt the ‘toothed’ feature values when hair = TRUE
Gini Index (hair = true & toothed = true) = 1 — {(4/4)2 + (0/4)2} = 1–1–0 = 0
Gini Index (hair = true & toothed = false) = 1 — {(1/1)2 + (0/1)2} = 1–1–0 = 0
To obtain the final Gini index of ‘species’ wrt ‘toothed’ for ‘hair=true’, we use the weighted sum of the above calculated values as:
Gini Index (toothed & hair = true) = (4/5) * 0 + (1/5) * 0 = 0 + 0 = 0
(ii) breathes: Let’s summarize the output (species) wrt the ‘breathes’ feature values when hair = TRUE
Gini Index (hair = true & breathes = true) = 0
Gini Index (hair = true & breathes = false) = 0
To obtain the final Gini index of ‘species’ wrt ‘breathes’ for ‘hair=true’, we use the weighted sum of the above.
Gini Index (breathes & hair = true) = 0
(iii) legs: Let’s summarize the output (species) wrt the ‘legs feature values when hair = TRUE
Gini Index (hair = true & legs = true) = 0
Gini Index (hair = true & legs = false) = 0
To obtain the final Gini index of ‘species’ wrt ‘legs’ for ‘hair=true’, we use the weighted sum of the above.
Gini Index (legs & hair = true) = 0
In all the above cases, the Gini Index is equal to 0. This means that if we predict that the species has hair, it can be classified as a ‘mammal’ based on the given dataset with high confidence. At this point, the decision tree looks like:
Now, let’s consider the case when ‘hair’ takes the values of FALSE.
Let’s calculate Gini Index for the output against the input features excluding the hair feature, when hair takes the value of FALSE.
(i) toothed: Let’s summarize the output (species) wrt the ‘toothed’ feature values when hair = FALSE
Gini Index (hair = false & toothed = true) = 1 — {(1/4)2 + (3/4)2} = 1–0.06–0.56 = 0.38
Gini Index (hair = false & toothed = false) = 1 — {(0/1)2 + (1/1)2} = 1–0–1 = 0
To obtain the final Gini index of ‘species’ wrt ‘toothed’ for ‘hair=false’, we use the weighted sum of the above calculated values as:
Gini Index (toothed & hair = false) = (4/5) * 0.38 + (1/5) * 0 = 0.3 + 0 = 0.3
(ii) breathes: Let’s summarize the output (species) wrt the ‘breathes’ feature values when hair = FALSE
Gini Index (hair = false & breathes = true) = 1 — {(1/4)2 + (3/4)2} = 1–0.06–0.56 = 0.38
Gini Index (hair = false & breathes = false) = 1 — {(0/1)2 + (1/1)2} = 1–0–1 = 0
To obtain the final Gini index of ‘species’ wrt ‘breathes’ for ‘hair=false’, we use the weighted sum of the above calculated values as:
Gini Index (breathes & hair = false) = (4/5) * 0.38 + (1/5) * 0 = 0.3 + 0 = 0.3
(iii) legs: Let’s summarize the output (species) wrt the ‘legs’ feature values when hair = FALSE
Gini Index (hair = false & legs = true) = 1 — {(1/2)2 + (1/2)2} = 1–0.25–0.25 = 0.5
Gini Index (hair = false & legs = false) = 1 — {(0/3)2 + (3/3)2} = 1–0–1 = 0
To obtain the final Gini index of ‘species’ wrt ‘legs’ for ‘hair=false’, we use the weighted sum of the above calculated values as:
Gini Index (toothed & hair = false) = (2/5) * 0.5 + (3/5) * 0 = 0.2 + 0 = 0.2
In the above 3 cases, we see that the Gini Index is minimum for the ‘legs’ feature given the condition ‘hair=FALSE’. So, ‘legs’ is chosen as the child node when ‘hair=FALSE’. At this point, the decision tree looks like:
To proceed further, let us consider the cases when legs = TRUE (hair = FALSE is implied now).
Let’s calculate Gini Index for the output against the input features excluding the hair and legs feature, when hair takes the value of FALSE and legs takes the value of TRUE.
(i) toothed: Let’s summarize the output (species) wrt the ‘toothed’ feature values when hair = FALSE and legs = TRUE.
Gini Index (hair = false & legs = true & toothed=true) = 0
Gini Index (hair = false & legs = true & toothed=false) = 0
To obtain the final Gini index of ‘species’ wrt ‘toothed’ for ‘hair=false’ and legs=’true’, we use the weighted sum of the above.
Gini Index (toothed & hair = false & legs = true) = 0
(ii) breathes: Let’s summarize the output (species) wrt the ‘breathes’ feature values when hair = FALSE and legs = TRUE.
Gini Index (hair = false & legs = true & breathes=true) = 1 — {(1/2)2 + (1/2)2} = 1–0.25–0.25 = 0.5
Gini Index (hair = false & legs = true & breathes=false) = 0
To obtain the final Gini index of ‘species’ wrt ‘toothed’ for ‘hair=false’ and legs=’true’, we use the weighted sum of the above.
Gini Index (toothed & hair = false & legs = true) = (2/2) * 0.5 + (0/2) * 0 = 0.5
In the above 2 cases, we see that ‘toothed’ has lower Gini Index, so it is selected as a child node when ‘legs = TRUE’. The decision tree now looks like:
Now, let’s consider the case when ‘legs’ takes the value of FALSE.
From the above scenario, we can clearly see that when ‘hair = TRUE’ and ‘legs = FALSE’, the output ‘species’ is ‘Reptile’ in all the cases. Hence, we without calculating the Gini Index, we can safely conclude that the child node produced for the decision tree will be a leaf node with the value ‘Reptile’. (Remember that when we just considered ‘hair = TRUE’ case, we saw that all the output ‘species’ values were ‘Mammals’ and by calculating Gini Index, we directly produced a leaf node with the value ‘Mammal’)
Proceeding further, let’s consider the case when ‘toothed = TRUE’. Here, the cases ‘hair = FALSE’ and ‘legs = TRUE’ are implied.
Here, we only have one record which is classified as mammal. Hence, this combination of features results in the output ‘species’ being a ‘Mammal’.
Considering the case ‘toothed = FALSE’, we get
In a similar way, we can conclude that the output ‘species’ for this set of features is a ‘Reptile’. Now, we have finally constructed the decision tree on which we can throw some test data to make predictions.
If you’ve followed carefully so far, you may have observed that all the variables in this dataset were categorical variables (i.e. the features had a finite set of distinct values (2) like TRUE and FALSE). In the datasets that you encounter in real life have a mixture of categorical and continuous variables. In order to construct decision trees from a dataset having continuous values, these continuous values are converted to categorical values by defining a certain threshold. Using this, decision trees can be constructed for any type of data.
Although decision trees are quite intuitive and simple to create, they suffer from the following drawbacks.
1. They are unstable, meaning that a small change in the data can lead to a large change in the structure of the optimal decision tree.
2. Calculations can get very complex, particularly if many values are uncertain and/or if many outcomes are linked.
3. Decision Trees tend to overfit to the training data very quickly and can become very inaccurate.
Due to the aforementioned disadvantages of Decision Trees, Random Forest algorithm is used instead of Decision Trees. Random Forest algorithm employs the use of a large number of decision trees that operate as an ensemble. A machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.
Random Forest algorithm is based on the concept of The Wisdom of Crowds. Wisdom of Crowds is a very effective technique employed in various tasks of computer science. The fact that Wikipedia (fully crowdsourced, editable by everyone) and Britannica (fully written by experts) have been found to have similar levels of quality standards is a testament to the power of the wisdom of crowds. Every individual in the crowd makes their prediction with a certain error. Since all the individuals act independently, their errors are also independent to each other’s. In other words, these errors are uncorrelated i.e. when considered in a bunch, these errors cancel each other out and what we’re left with is a good, accurate prediction. This is the way how Random Forest algorithm operates.
Random Forest algorithm uses a large number of decision trees (creating a forest), which are independent of/uncorrelated to each other, from a given dataset, which collectively outperform any of its constituent trees. This little to no correlation between the constituent trees in the forest imparts the element of randomness to this ensemble method, which produces a wonderful effect of protecting trees from each other for their individual errors. But we need to make sure that the outputs of the various decision trees that we make randomly for random forest to work should not be completely random. In order to ensure this, we need to make sure that:
(i) The output should be dependent on all the training features and not only some of them in order to make random decision trees effective.
(ii) The predictions made by each decision tree should have low correlations with each other in order to make the wisdom of crowds concept to work.
The first point is the feature of the dataset that we collect. So, as a data scientist, how do we make sure that the constituent decision trees are as diversified as possible? For this, random forest uses the following 2 techniques:
(i) Bagging
(ii) Feature Randomness
One of the major drawbacks of Decision Trees was that they were not robust to even small changes in the data which made them prone to overfitting. Let’s understand how bagging solves this problem. Consider a training dataset with M ( >> 50) training examples and for simplicity let’s consider these are M numbers in the range of 1 to 50. If we want to use B decision trees for Random Forest, then through bagging, each of the B decision trees is made up of M training examples, but instead of using all the M training examples, a random subset of these training examples (here numbers in the range from 1 to 50) is used and numbers from this subset are randomly repeated until a total of M training examples are created.
How does bagging help? By creating a large number of decision trees with a large number of training examples distributed randomly across decision trees, the aggregate result is a random forest model which is robust to even small changes in the data, thereby eliminating one of the major disadvantages of decision trees (by using decision trees).
The idea of using random training examples can also be applied to the features of the training data. Recall that while constructing decision trees, when deciding the split at each node, we take all the features into account. If the training dataset has N features, then in feature randomness, while creating each of the B decision trees, the decision of splitting a node is made by using a random subset of these N features instead of using all N features.
When both bagging and feature randomness are used to make a forest of decision trees for random forest, each of the decision trees constitutes of a random subset of training examples as well as a random subset of features. This randomness adds a lot of robustness and stability to a Random Forest classifier by ensuring the formation of uncorrelated decision trees that protect each other from their errors. Once all these constituent decision trees make a prediction, majority voting takes place in order to get the final prediction of the classifier.
We’ve looked at a lot of concepts so far. From the mathematics behind decision trees to creating a decision tree by hand and then seeing how decision trees are used in random forest, we’ve come a long way. Now is the time that we get hands on practice for a Random Forest classifier and for this post, we won’t be dealing with a new dataset, instead we’ll be improving upon our Logistic Regression model which we trained over the Titanic dataset in this post. This is the way actual machine learning tasks are carried out. Various machine learning models are tried and the one that works the best and justifies the results obtained is used in practice. Remember, persistence is the key. So let’s get started.
The dataset for the Titanic problem can be found here. We have to predict whether a passenger on the Titanic will survive or not given the data corresponding to them. We have already applied data pre-processing and logistic regression in this post. Now, we will use the same pre-processed data from the previous post, but will apply Random Forest classification this time. Dataset after pre-processing looks like:
After applying recursive feature elimination(RFE), we obtain the top 8 most important features that will help in training the Random Forest Classifier. More on RFE can be found here.
from sklearn.ensemble import RandomForestClassifierfrom sklearn.feature_selection import RFEcols = [“Age”, “SibSp”, “Parch”, “Fare”, “Pclass_1”, “Pclass_2”, “Pclass_3”, “Embarked_C”, “Embarked_Q”, “Embarked_S”, “Sex_male”]X = final_train[cols]y = final_train[‘Survived’]model = RandomForestClassifier()# selecting top 8 featuresrfe = RFE(model, n_features_to_select = 8)rfe = rfe.fit(X, y)print(‘Top 8 most important features: ‘ + str(list(X.columns[rfe.support_])))
Now that we have the list of features we should use for training the classifier, let us apply Random Forest classifier on the dataset. Unlike Logistic Regression, Random Forest has a large number of parameters that need to be fed into the model. The most important parameters are:
1. n_estimators = number of trees in the foreset
2. max_features = max number of features considered for splitting a node
3. max_depth = max number of levels in each decision tree
4. min_samples_split = min number of data points placed in a node before the node is split
5. min_samples_leaf = min number of data points allowed in a leaf node
6. bootstrap (like bagging) = method for sampling data points (with or without replacement)
These are a lot of parameters and it is difficult to obtain their optimal value together manually. So, we make use of the computation power by passing the model a set of values for each parameter, the parameter applies these values to the dataset and chooses the best ones. For this dataset, this took me about 5 minutes to get optimal values using Random Hyperparameter Grid technique.
# Number of trees in random forestn_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]# Number of features to consider at every splitmax_features = ['auto', 'sqrt']# Maximum number of levels in treemax_depth = [int(x) for x in np.linspace(10, 110, num = 11)]max_depth.append(None)# Minimum number of samples required to split a nodemin_samples_split = [2, 5, 10]# Minimum number of samples required at each leaf nodemin_samples_leaf = [1, 2, 4]# Method of selecting samples for training each tree (bagging)bootstrap = [True, False]# Creating random gridrandom_grid = {'n_estimators': n_estimators, 'max_features': max_features, 'max_depth': max_depth, 'min_samples_split': min_samples_split, 'min_samples_leaf': min_samples_leaf, 'bootstrap': bootstrap}
On each iteration, the algorithm will choose a difference combination of the features. Altogether, there are 2 * 12 * 2 * 3 * 3 * 10 = 4320 settings! However, the benefit of a random search is that we are not trying every combination, but selecting at random to sample a wide range of values.
# First create the base model to tunerf = RandomForestClassifier()# Random search of parameters, using 3 fold cross validation,# search across 100 different combinations, and use all available coresrf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)# Fit the random search modelrf_random.fit(final_train[selected_features], final_train['Survived'])rf_random.best_params_
Now that we have an optimal set of parameters from the given range supplied to the algorithm, let’s train the random forest classifier, both with default values that sci-kit learn provides and with the values we obtained.
X = final_train[selected_features]y = final_train['Survived']X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=2)# fitting both base random forest model and RandomizedSearchCV random forest modelbase_model = RandomForestClassifier()base_model.fit(X_train, y_train)best_random = RandomForestClassifier(n_estimators = 2000, min_samples_split= 5, min_samples_leaf = 1, max_features = 'sqrt', max_depth = 100, bootstrap = True)best_random.fit(X_train, y_train)
Now that we have trained these models, let’s evaluate them using the evaluation function defined below.
def evaluate(model, test_features, test_labels):  predictions = model.predict(test_features)  errors = abs(predictions - test_labels)  accuracy = accuracy_score(test_labels, predictions) * 100  print('Model Performance')  print('Average Error: {:0.4f} degrees.'.format(np.mean(errors)))  print('Accuracy = {:0.2f}%.'.format(accuracy))  return accuracybase_accuracy = evaluate(base_model, X_test, y_test)
random_accuracy = evaluate(best_random, X_test, y_test)
We observe that the base model had a test accuracy of 84.33% whereas the model in which we fed the parameters obtained RandomizedSearchCV yielded test accuracy of 81.34%, which should have been higher than the base model’s accuracy. This can be due to the fact that the training dataset was not big enough or the values we supplied for random search to choose from might have included the range of optimal values. We can definitely toy around with it to improve our accuracy though.
Similarly, I also applied GridSearchCV to check for the accuracy of the classifier which was again around 81%. The relatively low accuracy might be due to the reasons posited above.
But, the key takeaway from this exercise is that we improved our model’s accuracy on the test set from 77.6% using Logistic Regression to around 84% using Random Forest algorithm, which indeed justified our discussion of Random Forest algorithm above. When I submitted the results of this model on Kaggle, I received a public score of 0.7751, which is better than the score we received for Logistic Regression classifier (0.7703), although not by much, but good enough to take a step forward to improve an already existing robust classifier.
The entire code for this post can be found here.
That’s it for this post. In all the posts so far, we have explored the most frequently used supervised machine learning algorithms in great detail. From the next post onwards, we’ll dive deep into the basics of Unsupervised Learning and also learn about various unsupervised learning algorithms, starting with K-means clustering.