Dealing with Multiclass Data. Forest Cover Type Prediction | by Amber Teng | Towards Data Science

Have you ever thought about what to do when you encounter a classification problem that consists of over three classes? How did you deal with multiclass data, and how did you evaluate your model? Was overfitting a challenge — and if so, how did you surmount that?
Read on to discover how I worked through these questions on my latest project — analyzing forest cover types from a dataset provided by the US Forest Service, and housed on the UCI Machine Learning Repository.
This dataset is specifically interesting because it consists of a mix of both categorical and continuous variables, which has historically required different techniques of analysis. These variables describe the geology of each sample forest region, and a multiclass label (one of seven possible tree cover types) serves as our target variable.
According to Kaggle, these seven possible cover types are the following:
Spruce/FirLodgepole PinePonderosa PineCottonwood/WillowAspenDouglas-firKrummholz
Spruce/Fir
Lodgepole Pine
Ponderosa Pine
Cottonwood/Willow
Aspen
Douglas-fir
Krummholz
I thought that this was a cool application of what I’ve learned during my fellowship at SharpestMinds because successful forest cover type classification has so much potential for positive change, particularly in areas like environmental conservation, flora and fauna research, and geological studies.
This Medium blogpost will focus on tree-based methods I explored to analyze this dataset, but you can view my whole repo, which contains other machine learning methods, here: https://github.com/angelaaaateng/Covertype_Analysis
The first step is exploratory data analysis. I loaded the data and read the information into a dataframe using pandas.
Then, I checked the data types to ensure that there were no anomalies in my dataset — and to get a better view of what kind of data I was dealing with. Based on this, we also saw that there were no NaN values, the columns were sensible, and in general, the dataset was pretty clean and didn’t need additional cleaning at this stage and for the goals of this project.
We also see that we’re dealing with a pretty large dataset, so it’s highly likely that we’ll need to downsample from the 581,012 entries. We also know that we have 55 columns or features, which will be explored more thoroughly in the next section. Next, let’s take a look at the frequency distributions of these forest cover types.
The bulk of the dataset is comprised of covertype 1 and covertype 2, and with a dataset that is this imbalanced, we have to make a choice about the way we downsample or build our toy dataset. The key lies in understanding the answer to this question: will we aim to have equal representation of all covertypes, or will we aim to represent each covertype in proportion to its frequency in the overall dataset?
One important consideration is that downsampling does reduce the number of data points we can use to train our model, so despite this analysis, it’s always possible that using all samples will be the wiser choice. This depends highly on business priorities, including the relative cost of mistaking one covertype for another. For example, if we’re equally concerned about classification accuracy for each covertype, then downsampling to get an even distribution of covertypes is likely to make the most sense. However, if we only cared about classifying cover type 1 versus the other 6 cover types, then we might use a different sampling method.
Nonetheless, for this project, I’ll start by assuming that we’re equally concerned with correctly classifying each covertype, and I’ll explore the impact of this decision towards the end of the post.
Looking more deeply into the statistics of each cover type, we see that there are big differences in count across samples, and we also see that there is a wide spread of data for each variable.
Particularly for elevation, we see that the standard deviation among covertypes 1 through 7 ranges from 95 to 196. We also see that for cover type 4, we have a minimum of 2,747 data points, while for cover type 2, we have a high of 28,8301, further showing that we have an unbalanced dataset.
We can explore the correlation between and among continuous variables to better understand underlying relationships in the data using a correlation matrix:
Note that I specifically left out the categorical variables in this correlation plot because the correlation matrix displays the Pearson correlation coefficients between different feature pairs, and Pearson correlations don’t make sense for non-continuous inputs. Based on the correlation matrices above, it seems like the continuous variables that are linearly related to covertype are slope and elevation, as well as aspect. It also appears that our three hillshade features are highly correlated to one another. This is to be expected, and does seem logical given that they all measure similar things. The correlation between these hillshades suggests that they may contain a fair amount of redundant information — an idea we’ll return to when we do feature selection.
Based on the data exploration we showed earlier, we saw that the target variable is multi-class and binary. To determine what a “good” model looks like, and how we’re defining success in this project, I first looked into establishing a baseline model.
However, before we can go about training a multiclass baseline model, we need to sample the data first. Not only will it be time consuming to train a model on all 500,000+ of our entries, it is also very inefficient.
For our particular dataset, we’ll begin with a simple randomized sampling method, where we ensure a balanced class distribution, for the reasons outlined at the beginning of this post. This means that we want to take a sample of n entries from each covertype from 1 to 7, and ensure that we have the same number of entries per class.
We’ll start with n = 2700, which is the minimum number of data points for the smallest class, and we’ll begin by using a simple decision tree to model our data. Our goal in doing this will be to understand our data better, by studying how our decision tree ends up interrogating our dataset, and to get a lower bound on our classification performance. This step puts us right at the fuzzy boundary between data exploration and predictive modeling, a grey zone that I’ve found particularly helpful to explore especially with tree-based models.
Let’s start by understanding our data a bit better. I started by fitting a very short decision tree to the downsampled dataset (using a max_depth of 3), and visualized the result:
The first thing we’ll note from this is the importance of the elevation parameter, which appears to dominate all others in this admittedly simple model. This is consistent with what we found in the correlation matrix earlier, which showed that elevation was the continuous feature that was most correlated with our target variable (Cover_Type).
Although this model’s max_depth is severely constraining, it still performs much better than random guessing: the predictive accuracy it gets across all 7 classes is X% (training accuracy) and Y% (validation accuracy).
Next, I wanted to see how an unconstrained model (without a max_depth of 3) would perform, to get a better idea of the full predictive power of a simple decision tree on this dataset. I trained a new tree with the default parameters provided by sklearn (which includes a max_depth of None, meaning that the tree will grow without limit). Here were the results:
We’ll be using accuracy score as a measure of performance.
predictions = dtree.predict(X_test)print ("Decision Tree Train Accuracy:", metrics.accuracy_score(y_train, dtree.predict(X_train)))print ("Decision Tree Test Accuracy:", metrics.accuracy_score(y_test, dtree.predict(X_test)))from sklearn.metrics import classification_reportprint(classification_report(y_test, y_pred))
We can see that there is overfitting here, as the accuracy score of the train data is 1. This is an important result because it is consistent with the general trend of decision trees tending to overfit. For fun, I thought I’d show exactly how this overfitting process plays out by plotting training and validation accuracies as a function of the max_depth parameter:
The above graph shows us that the testing accuracy is less than the training accuracy, which is the definition of overfitting: our model is twisting itself into a shape that perfectly captures the trends in the training set, at the expense of being able to generalize to unseen data.
To wrap up our data exploration-and-initial modeling step, let’s take a look at the confusion matrix that results from applying our deep tree to a validation set:
Looking at the confusion matrix above, it seems that most of these errors are coming from classes 1 and 2.
Now that we know what to expect from our problem, it’s time to get serious about modeling. Random forests are the model I’ll use here, because 1) they’re robust, and generalize well; and 2) they’re readily interpretable. Random forests are ensembles of decision trees: they consist of a bunch of independent decision trees, each of which is trained using only a subset of the features in our training set to ensure that they’re learning to make their predictions in different ways. Their outputs are then pooled together using simple voting.
As always, my first step was to use an out of the box random forest classifier model. This resulted in a big bump in performance: 86% accuracy on the validation set, and 100% accuracy on the training set. In other words, the model is overfitting (or rather, each decision tree in the ensemble is overfitting) but we’re nonetheless seeing a big improvement in performance from pooling together a bunch of overfit decision trees.
First, let’s do feature selection to identify the most predictive variables that affect the accuracy rate of our random forest ensemble model. People use many different methods for this, but here, we’ll focus on permutation feature importance. Permutation feature importance works by selecting a column (i.e. feature) in our validation set, then shuffling it randomly, thereby destroying the correlations between that feature and all the other features used by our model to make its predictions, and finally measuring our model’s performance on this freshly shuffled validation set. If the performance drops significantly, that tells us that the feature we shuffled must have been important.
Using permutation importance, we see the following results:
Based on this, it really seems like only a small number of our features dominate the others. In fact, if when I tried keeping only the top 13 features indicated here, I ended up sacrificing only a negligible amount of accuracy, obtaining once again 86% accuracy on the validation set.
One key to hyperparameter tuning in random forests is that in general, the model’s performance only increases with the number of decision trees that we add to our ensemble. As a result, it’s actually the last parameter we’ll tune, after we’ve finished tuning all the other relevant parameters (like max_depth, min_samples_leaf and min_samples_split) using GridSearchCV.
We get the final set of best estimators, and, given these best parameters, we then apply these to our model and compare the results:
n_best_grid = n_grid_search.best_estimator_n_optimal_param_grid = {    'bootstrap': [True],    'max_depth': [20], #setting this so as not to create a tree that's too big    #'max_features': [2, 3, 4, 10],    'min_samples_leaf': [1],    'min_samples_split': [2],    'n_estimators': [300]}nn_grid_search = GridSearchCV(estimator = n_rfc_gs, param_grid = n_optimal_param_grid,                           cv = 3, n_jobs = -1, verbose = 2)nn_grid_search.fit(X_train, y_train)nn_rfc_pred_gs = nn_grid_search.predict(X_test)nn_y_pred_gs =  nn_grid_search.predict(X_test)print ("Random Forest Train Accuracy Baseline After Grid Search and N-estimators Search:", metrics.accuracy_score(y_train, nn_grid_search.predict(X_train)))print ("Random Forest Test Accuracy Baseline After Grid Search and N-estimators Search:", metrics.accuracy_score(y_test, nn_grid_search.pre
Our model has performance has improved modestly, from 0.860 to 0.863. This isn’t entirely surprising — hyperparameter tuning doesn’t always have a huge impact on performance.
Finally, we’ll take a look at the num_estimators parameter, taking care to plot the training and validation set accuracies together. As expected, both increase approximately monotonically with num_estimators, until reaching a plateau:
We can then compare our current models and their performance based on accuracy, and compare it to the original out of the box models.
One important point to discuss here is how in the initial run (not shown in this blogpost but in the notebook here), our testing accuracy went from 86% without parameter search to 84% after using GridSearchCV. We can see that the tuned model performs no better than the RF trained on default values. That’s because of the effect of variance on our dataset — GridSearch struggles to differentiate between the best hyperparameters and the worse ones, so it might save time and computing power to go with the out of the box random forest model.
Some important possibilities to think about in relation to this point are the following:
We are using GridSearch with cv = 3 (3-fold cross-validation). That means that for every hyperparameter combination, the model gets trained on only 2/3 of the data in the training set (because 1/3 is kept for validation). So based on this, we’d expect GridSearch to produce a more pessimistic result. One way to avoid this is to increase the value of cv so that you have smaller validation sets and larger training sets.
There’s always noise (or “variance”) to consider: when we train our model on a different dataset, it will in general give different results. If the variance of our model is high enough (meaning that its performance depends heavily on the particular points that it’s trained on) then it’s possible that GridSearch actually can’t differentiate between the “best” hyperparameters and the worst ones. This is the most likely explanation for our current results. This means that we might as well just go with the default values.
Another area we can more deeply explore is max_features.
So, how do we do better? Based on our confusion matrices, it appears that classes 1 and 2 are where most of the error is coming from. In my next project, we will explore model stacking to see if we can gain any accuracy here.
For any questions or comments, feel free to reach out to me on Twitter @ambervteng Thanks for reading, and see you next time!
Github Repo: https://github.com/angelaaaateng/Covertype_Analysis
Heroku Web App: https://covertype.herokuapp.com/
Jupyter Notebook: https://github.com/angelaaaateng/Projects/blob/master/Covertype_Prediction/Scripts/Tree-Based%20and%20Bagging%20Methods.ipynb
[1] DataCamp Decision Tree Tutorial https://www.datacamp.com/community/tutorials/decision-tree-classification-python
[2] Seif, G. Decision Trees for ML https://towardsdatascience.com/a-guide-to-decision-trees-for-machine-learning-and-data-science-fe2607241956
[3] Matplotlib Documentation https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.xticks.html