A Practical Guide to Implementing a Random Forest Classifier in Python | by Eden Molina | Towards Data Science

Random forest is a supervised learning method, meaning there are labels for and mappings between our input and outputs. It can be used for classification tasks like determining the species of a flower based on measurements like petal length and color, or it can used for regression tasks like predicting tomorrow’s weather forecast based on historical weather data. A random forest—as the name suggests—consists of multiple decision trees each of which outputs a prediction. When performing a classification task, each decision tree in the random forest votes for one of the classes to which the input belongs. For example, if we had a dataset on flowers and we wanted to determine the species of a flower, the decision trees in a random forest will each cast a vote for which species it thinks a flower belongs. Once all the trees have come to a conclusion, the random forest will count which class (species) had the most populous vote and this class will be what the random forest outputs as a prediction. In the case of regression, instead of determining the most populous vote the random forest will average the results of each decision tree. Because random forests utilize the results of multiple learners (decisions trees), random forests are a type of ensemble machine learning algorithm. Ensemble learning methods reduce variance and improve performance over their constituent learning models.
As mentioned above, random forests consists of multiple decision trees. Decision trees split data into small groups of data based on the features of the data. For example in the flower dataset, the features would be petal length and color. The decision trees will continue to split the data into groups until a small set of data under one label ( a classification ) exist. A concrete example would be choosing a place to eat. One might peruse Yelp and find a place to eat based on key decisions like: is the place open right now, how popular is this place, what is the average rating for this place, and do I need a reservation. If the place is open, then the decision tree will continue on to the next feature if not the decision ends and the model outputs “no don’t eat here”. Now at the following feature, the model will determine if a place is popular, if the location is popular the model will move on to the next feature if not the model will output “no don’t eat here”—same procedure as before. Essentially one can think of a decision tree as a flowchart mapping out decisions once can take based on data until a final conclusion is made. The decision tree determines where to split the features based on a purity measure that measures information gain. In the case of classification it makes that decision based on the Gini index or entropy and in the case of regression, the residual sum of squares. However, the mathematics of this is beyond the scope of this blog post. In essence what these metics measure is Information Gain, the measure of how much information we’re able to gather from a piece of data.
The random forest algorithm can be described as follows:
Say the number of observations is N. These N observations will be sampled at random with replacement.Say there are M features or input variables. A number m, where m < M, will be selected at random at each node from the total number of features, M. The best split on these m variables are used to split the node and this value remains constant as the forest grows.Each decision tree in the forest is grown to its largest extent.The forest will output a prediction based on the aggregated predictions of the trees in the forest. (Either majority vote or average)
Say the number of observations is N. These N observations will be sampled at random with replacement.
Say there are M features or input variables. A number m, where m < M, will be selected at random at each node from the total number of features, M. The best split on these m variables are used to split the node and this value remains constant as the forest grows.
Each decision tree in the forest is grown to its largest extent.
The forest will output a prediction based on the aggregated predictions of the trees in the forest. (Either majority vote or average)
In this post we will be utilizing a random forest to predict the cupping scores of coffees. Coffee beans are rated, professionally, on a 0–100 scale. This dataset contains the total cupping points of coffee beans as well as other characteristics of the beans such as country of origin, variety, flavor, aroma etc. Some of these features will be used to train a random forest classifier to predict the quality of a particular bean based on the total cupping points it received. The data in this demo comes from the TidyTuesday Repository and below is a preview of what the data looks like.
To keep the model safe we will subset this dataset to contain columns that are attributes of the coffee which are, but not limited to: species, country of origin, acidity, body, and sweetness. Furthermore, we will drop any rows that contain nan (not a number/missing) values.
coffee_df = coffee_ratings[['total_cup_points',                            'species',                            'country_of_origin',                            'variety',                            'aroma',                            'aftertaste',                            'acidity',                            'body',                            'balance',                            'sweetness',                            'altitude_mean_meters',                            'moisture']]coffee_df = coffee_df.dropna()
From above we can see the data contains not only numeric values but categorical values as well like species, country of origin, and variety. We can map these out to numeric values by using the OrginalEncoder function provided by the sklearn package. For example there are two species of coffee: Arabica and Robusta. The code will assign numerical values to these such that Arabica maps to 0, and Robusta maps to 1. A similar logic will follow for the other variables.
from sklearn.preprocessing import OrdinalEncoderord_enc = OrdinalEncoder()coffee_df["species"] = ord_enc.fit_transform(coffee_df[["species"]])coffee_df["country_of_origin"] = ord_enc.fit_transform(coffee_df[["country_of_origin"]])coffee_df["variety"] = ord_enc.fit_transform(coffee_df[["variety"]])
Now that all the data is numeric we can generate charts to observe the relations between the different features. A useful chart to make is a pairplot which shows the marginal and joint distributions of the features. We drop the
import seabornseaborn.pairplot(coffee_df.drop('total_cup_points', axis = 1))
We can see the distributions are quite difficult to see especially. In some cases the distributions are quite binary such as speciies, or very skewed like average altitude. Furthermore, some numerical variables such as aroma and acidity contain one value each at 0, which are outliers in the data we will filter these out and replot the paitplot.
coffee_df = coffee_df[coffee_df['aroma']>0]coffee_df = coffee_df[coffee_df['acidity']>0]seaborn.pairplot(coffee_df)
Below we can see a clearer view of the joint distributions. The first column of plots represents how the features relate to the target variable total_cupping_points . We can see features from aroma to balance appear to have a strong relation to the total_cupping_points received. Furthermore, the marginal distributions, which we see on the diagonal of the pairplot appear to be, for the most part, normally distributed.
Another plot we can generate for exploratory data analysis (EDA) is a correlation matrix which will display the correlations between the features. We can also accomplish this with the package seaborn.
seaborn.heatmap(coffee_df.corr(),                xticklabels=coffee_df.columns,                yticklabels=coffee_df.columns)
We can see, a positive correlation between features from aroma to sweetness, and negative correlations with species to variety as well as altitude_mean_meters and moisture.
If we look at the distribution of total cupping ratings we can see the values are continuous and not discrete—as for example star ratings for restaurants, one star, two star, etc. In this project I will bin together the cupping ratings into three categories: low, average, and good. This can be accomplished by setting thresholds based on the distribution we see below. Anything within a certain threshold will be labeled as bad, average, or good. To be more precise, I will define anything below the 75th percentile to be labeled as ‘low’, anything between the 75th and 90th percentile as ‘average’, and the top 10 percentile as ‘good’. Of course, in the dataset, these labels will be numerical. Bad will be labeled as 1, average labeled as 2, and good labeled as 3.
This is how the thresholds will be computed. We will call the percentile function from numpy and input a list containing the percentiles represenitng the cutoff points.
rating_pctile = np.percentile(coffee_df['total_cup_points'], [75, 90])# The percentile thresholds arearray([83.58, 84.58])
We then use these thresholds to create a new column in the dataframe called n_rating representing the rating of the coffee using the new labeling system based on the percentile system defined above.
coffee_df['n_rating'] = 0coffee_df['n_rating'] = np.where(coffee_df['total_cup_points'] < rating_pctile[0], 1, coffee_df['n_rating'])coffee_df['n_rating'] = np.where((coffee_df['total_cup_points'] >= rating_pctile[0]) & (coffee_df['total_cup_points'] <= rating_pctile[1]), 2, coffee_df['n_rating'])coffee_df['n_rating'] = np.where(coffee_df['total_cup_points'] > rating_pctile[1], 3, coffee_df['n_rating'])
Below we can observe the new distribution of ratings given by out thresholds. The majority of coffee ratings are in the ‘low’ category (75% the data to be precise), and only a few coffees have a ‘good’.
At this point the data is nearly ready for modeling. We now need to split the data into training and testing sets. The training set will be used to train the random forest classifier, while the testing set will be used to evaluate the model’s performance—as this is data it has not seen before in training. Here 75% the data is used for training and 25% of the data is used for testing.
X = coffee_df.drop(['total_cup_points', 'n_rating', 'sweetness', 'species', 'altitude_mean_meters'], axis = 1)y = coffee_df['n_rating']training, testing, training_labels, testing_labels = train_test_split(X, y, test_size = .25, random_state = 42)
Next we will scale the data, this will being all our features to the same scale, this helps ensure attributes with larger values do not over influence the model. The StandardScaler in sklearn essentially computes the z score of each feature which ensures each feature in the dataset has a mean of 0 and variance of 1.
# Normalize the datasc = StandardScaler()normed_train_data = pd.DataFrame(sc.fit_transform(training), columns = X.columns)normed_test_data = pd.DataFrame(sc.fit_transform(testing), columns = X.columns)
For posterity we can plot another pairplot to see if normalizing accentuates the relations between the features. After normalizing the data, the relations of the coffee attributes from aroma to balance are clearer. Because of the elliptical joint distributions, there appears to be string relations between these vairables.
Now the data is prepped, we can begin to code up the random forest. We can instantiate it and train it in just two lines.
clf=RandomForestClassifier()clf.fit(training, training_labels)
Then make predictions.
preds = clf.predict(testing)
Then quickly evaluate it’s performance.
print (clf.score(training, training_labels))print(clf.score(testing, testing_labels))1.00.8674698795180723
The score method gives us an insight to the mean accuracy of the random forest on the given data. In the first call we evaluate it’s performance on the training data, then on the testing data. The model scores 100% accuracy on the training data and a lower 86.75% on the testing data. Since the training accuracy is so high and the testing is not as close, it’s safe to say the model is overfitting—the model is modeling the training data too well and not generalizing what it’s learning. Think of it as memorizing answers to a test instead of actually learning the concepts for that test. We will discuss how to to mitigate this issue later.
Another tool we can use to evaluate the performance of the model is by a confusion matrix. A confusion matrix shows the combination of the actual and predicted classes. Each row of the matrix represents the instances in a predicted class, while each column represents the instances in an actual class. It is a good measure of whether models can account for the overlap in class properties and understand which classes are most easily confused.
metrics.confusion_matrix(testing_labels, preds,  labels = [1, 2, 3])
The first column represents the coffees with a ‘low’ rating, the second column represents the coffees with an ‘average’ rating, and the third column represents the coffee with a ‘good’ rating. The numbers on the diagonal of the matrix, 183, 19, and 14 represent the number of coffees the model has accurately classified. The off diagonal values represent the misclassidied data. For example, 11 coffees which were supposed to be marked as ‘low’ scoring coffees were marked ‘average’.
Another interesting insight we can gain from a random forest is to explore what features the model ‘thinks’ are most important in determining the total cupping points.
pd.DataFrame(clf.feature_importances_, index=training.columns).sort_values(by=0, ascending=False)
We can see balance , aftertaste, and acidity appear to heavily contribute to the total cupping points a coffee receives.
When instantiating a random forest as we did above clf=RandomForestClassifier() parameters such as the number of trees in the forest, the metric used to split the features, and so on took on the default values set in sklearn. However, these default values more often than not are not the most optimal and must be tuned for each use case. The parameters to be tuned can be found in the sklearn documentation. One way to tune these parameters is to perform a random grid search. We define a range of values from which the code can randomly pick and choose until it finds a set that performs the best. To begin we define values the search algorithm will search through.
# Number of trees in random forestn_estimators = np.linspace(100, 3000, int((3000-100)/200) + 1, dtype=int)# Number of features to consider at every splitmax_features = ['auto', 'sqrt']# Maximum number of levels in treemax_depth = [1, 5, 10, 20, 50, 75, 100, 150, 200]# Minimum number of samples required to split a node# min_samples_split = [int(x) for x in np.linspace(start = 2, stop = 10, num = 9)]min_samples_split = [1, 2, 5, 10, 15, 20, 30]# Minimum number of samples required at each leaf nodemin_samples_leaf = [1, 2, 3, 4]# Method of selecting samples for training each treebootstrap = [True, False]# Criterioncriterion=['gini', 'entropy']random_grid = {'n_estimators': n_estimators,#                'max_features': max_features,               'max_depth': max_depth,               'min_samples_split': min_samples_split,               'min_samples_leaf': min_samples_leaf,               'bootstrap': bootstrap,               'criterion': criterion}
What this essentially creates is a dictionary of values to choose from:
{'n_estimators': array([ 100,  307,  514,  721,  928, 1135, 1342, 1550, 1757, 1964, 2171,        2378, 2585, 2792, 3000]), 'max_depth': [1, 5, 10, 20, 50, 75, 100, 150, 200], 'min_samples_split': [1, 2, 5, 10, 15, 20, 30], 'min_samples_leaf': [1, 2, 3, 4], 'bootstrap': [True, False], 'criterion': ['gini', 'entropy']}
Next we can begin the search and then fit a new random forest classifier on the parameters found from the random search.
rf_base = RandomForestClassifier()rf_random = RandomizedSearchCV(estimator = rf_base,                               param_distributions = random_grid,                               n_iter = 30, cv = 5,                               verbose=2,                               random_state=42, n_jobs = 4)rf_random.fit(training, training_labels)
We can also view the parameter values the random search found:
rf_random.best_params_{'n_estimators': 2171, 'min_samples_split': 2, 'min_samples_leaf': 2, 'max_depth': 200, 'criterion': 'entropy', 'bootstrap': True}
From here we can again evaluate the model
print (rf_random.score(training, training_labels))print(rf_random.score(testing, testing_labels))0.98259705488621150.8714859437751004
We now see the training score is less than 100%, and the testing score is 87.25%. The model doesn’t appear to be overfitting as much as before.
Now that we have parameter values from the randomized grid search, we can use these as a starting point for a grid search. A grid search works, in principle, similarly to the randomized grid search as it will search through the parameter space we define. However, instead of searching random values it will search every value possible in the parameter space. This computation can be expensive because the search grows very fast as more parameters and search values for those parameters increasess. This is why we performed the random grid search first. We save on compute power and only perform a fine methodological search once we have a good starting point. Instantiating a grid search is also similar to instantiating the random grid search.
param_grid = {    'n_estimators': np.linspace(2100, 2300, 5, dtype = int),    'max_depth': [170, 180, 190, 200, 210, 220],    'min_samples_split': [2, 3, 4],    'min_samples_leaf': [2, 3, 4, 5]}
We can then retrain yet another random forest.
# Base modelrf_grid = RandomForestClassifier(criterion = 'entropy', bootstrap = True)# Instantiate the grid search modelgrid_rf_search = GridSearchCV(estimator = rf_grid, param_grid = param_grid,                           cv = 5, n_jobs = 8, verbose = 2)grid_rf_search.fit(training, training_labels)
And view what it found to be the most optimal set of parameters
best_rf_grid = grid_rf_search.best_estimator_grid_rf_search.best_params_{'max_depth': 170, 'min_samples_leaf': 2, 'min_samples_split': 2, 'n_estimators': 2100}
We can see the results it found are not too far from what the random grid search found. Now we can evaluate this model to determine it’s accuracy and performance.
print (grid_rf_search.score(training, training_labels))print(grid_rf_search.score(testing, testing_labels))0.98393574297188760.8714859437751004
We see it performs very similarly to the random search grid as it’s parameters did not stray too far from what we initially started with.
Furthermore, we can also view the features the model thought were most importnact. Balance, aftertaste, and acidity are still the top 3 features with just slightly different weights.
Lastly we can view the confusion matrix for this final model. It appears it’s the same as before.
In this article, we performed some exploratory data analysis on the coffee dataset from TidyTuesday and built a Random Forest Classifier to classify coffees into three groups: low, average, good. These classes determine how many cupping points the coffee is expected to receive. Once we built an initial classifier we tuned it’s hyper-parameters to determine if we can further improve the performance of the mode.