Introduction to Random Forest Classifiers from sklearn | by Philip Wilkinson | Towards Data Science

Admittedly I am a huge fan of the NBA even though I am based in the UK so I don’t get to see much of the games. This means that I get my fix mostly from following the stats and the highlights after the games. Although I regret that I don’t get to watch as many games as I like, the analytical side of me enjoys being able to watch and follow the stats, usually being able to roll them off my tongue to any unsuspecting victim that engages me on the topic. Given this though, I thought it would be good to combine my love of basketball and analytics to integrate them into my learning of Data Science. One of my projects therefore is being able to predict the position of NBA players based on their stats in an attempt to answer the question of whether we are witnessing a truly ‘positionless’ league.
The data for this comes from the Basketball Reference website [1], which lists every NBA players stats for the 2018–2019 season. This season was used because of the limited games played in the 2019–2020 season which may affect any analysis that would have been performed. Data on each player included: Games played, minutes played, field goals, field goal attempts etc. which are typical of an NBA stat sheet. Of course, as part of any data analysis, the data had to be cleaned first. The data I extracted was the total stats for the season for all players and had some duplicates from when players were traded mid-season. The first thing to do was to remove these duplicates and convert the totals into per minute stats, so as to not allow difference in total amount influence the analysis. Of course, this may indicate efficiency rather than anything else (i.e. points per minute), but it was taken as the best non biased indicator of performance. This left me with 22 independent variables from which to identify position from, which, using a correlation plot, several variables were removed that were seen as highly correlated or may bias the results:
corrMatrix = NBA.corr()f, (ax) = plt.subplots(figsize=(15,15))sns.heatmap(corrMatrix, annot=True, ax=ax)plt.show()
Dropping columns of Field Goals, Field Goals attempted, Field Goal percentage because they were aggregates of the 2-point and 3-point variables, and free throw attempts, two point attempts and three point attempts were removed as these could also be represented in free throw percentage, 2 point percentage and three point percentage.
After that, the analysis could proceed. This included splitting the data into our independent variables and the dependent variable and creating a train test split so that the model could be validated.
X = NBA.drop(columns = ["Pos"])y = NBA.Posfrom sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state=42)
The Random Forest Classifier algorithm is an ensemble method in that it utilises the Decision Tree Classifier method but instead of creating just a single Decision Tree, multiple are created. In doing so, it takes advantage of random sampling of the data as each tree learns from a random sample of the data points which are drawn without replacement, and uses a subset of the features considered when splitting nodes. This randomness in generating individual trees minimises the potential for over-fitting and improves the overall predictive accuracy of the model. This is because the final predictions are made by averaging the predictions of each individual tree, thus following the logic that the performance of the crowd is better than the performance of the individual.
The advantages of this are that:
The potential for overfitting is removed by the randomness
The results can be seen as highly accurate and robust
We can extract feature importance, which tells us which variables contribute to the results the most
Of course, there are also disadvantages:
The model can take longer than any decision tree because of the increased complexity
The model may be difficult to interpret in comparison to a single decision tree because of the combination of many decision trees
For this application however the Random Forest Algorithm is a suitable algorithm because of its complexity, its ability to be highly accurate and robust, and the potential to extract feature importance given the high number of independent variables included in the model.
In using the Random Forest Classifier we also want to test the results against a baseline, which we can do by creating a dummy classifier which makes decisions based on simple rules, such as putting all players into the largest category, which in this case is the shooting guard position:
from sklearn.dummy import DummyClassifierfrom sklearn.metrics import accuracy_scorefrom sklearn import metricsdc = DummyClassifier(strategy='most_frequent')dc.fit(X_train,y_train) dc_preds = dc.predict(X_test)print (metrics.classification_report(y_test, dc_preds))
Now, we actually want to generate the model for our data and see how it compares. First thing is to therefore import the Random Forest Classifier algorithm, taken from the sklearn.ensemble module. There are a variety of parameters for this that could be altered depending on what we want from our decision tree, with explanations from here and here. For our purpose, we will still with the default gini criterion for splitting, set the number of estimates at 100 to create 100 individual decision trees, set the oob_score as True and set the max_depth as 3 so that we can later visualise some of the individual trees:
#import the classifierfrom sklearn.ensemble import RandomForestClassifierclf = RandomForestClassifier(n_estimators =100, oob_score=True, max_depth =3)#fit it to the training dataclf.fit(X_train, y_train)#extract the predictionstest_pred_random_forest = clf.predict(X_test)
Once this is fitted, we can then generate the confusion matrix based on the predictions generated by the model. Here, the true label is given on the Y axis, while the predicted label is given on the X-axis:
# get the confusion matrixconfusion_matrix2 = metrics.confusion_matrix(y_test, test_pred_random_forest)#conevrt the matrix to a dataframematrix2_df = pd.DataFrame(confusion_matrix2)# set axis to add title and axis labels laterax = plt.axes()sns.set(font_scale=1.3) # for label sizeplt.figure(figsize=(12,12))plot = sns.heatmap(matrix2_df, annot=True, fmt='g', ax=ax, cmap = "magma") #fmt so that numbers aren't scientific#axis labels and titleax.set_title('Confusion Matrix - Random Forest', fontsize = 17)ax.set_xlabel('Predicted Label', fontsize = 15)ax.set_ylabel('True Label'     , fontsize = 15)# change tick labels from 0-4 to 1-5labels = ("C", "PF", "PG", "SF", "SG")ax.set_xticklabels(labels)ax.set_yticklabels(labels)plt.show()fig = plot.get_figure()
From this we can see that the model has done relatively well at predicting Centres and Power Forwards with the main issues being predicting that they are each other, with small forwards often being confused with power forwards and some confusion between point guards and shooting guards. Of course, we can quantify this through the precision, recall, f1-score and accuracy metrics as we did for the dummy classifier:
print (metrics.classification_report(y_test, test_pred_random_forest))
For descriptions of each metric please see here, but we can see that compared to the dummy classifier, the overall accuracy of the model has greatly increased from 0.28 to 0.63 indicating that our model is better than a simple decision rule. However, with only a 63 percent accuracy we are still incorrectly predicting 37 percent, which suggests that our model is not perfect. We can therefore consider how we may be able to improve our model fit.
What is good about the Random Forest Classifier, as compared to a single Decision Tree, is that we can examine the feature importance more closely. This is because while a single decision tree may focus on a particular part of the data and thus associate high importance with that feature even if it doesn’t necessarily have high importance, random sampling and feature selection across multiple decision trees allows us to identify which features consistently have high importance. By creating multiple trees, we can use the average predictive ability of the features in order to reduce the variance that would be associated with feature importance if we were using an individual tree.
From this understanding, we can remove features that are not important and re-run the models again, or use the information simply to understand which features are not important for the model. Removing features will reduce the complexity of the model and thus reduce the compute time, so the trade off is the information gained from features against compute time.
fi = pd.DataFrame({'feature': list(X.columns),                  'importance': clf.feature_importances_}).\                    sort_values('importance', ascending=True)plot = fi.plot.barh(x = 'feature', y = 'importance', figsize=(15,15))fig = plot.get_figure()
Our results show that 6 features are highly informative while the remaining 11 are less so. The decision to remove these features, if at all, is often based on a threshold (such as 1%) or we could compare them against a random feature (i.e. random set of numbers) which if they perform worse than they would be removed as they simply represent noise.
The alternative measure for this is the permutation importance, which uses random re-shuffling to see how each individual variable influences the model outcome. This does so by training the baseline model, recording the score (in this case the R2 score, then reshuffling the values from one feature in the dataset and rerun the predictions. The feature importance here is the difference between the benchmark score and the modified score, thus the higher the value the more important the feature is for the model:
from sklearn.metrics import r2_scorefrom sklearn.inspection import permutation_importanceresult = permutation_importance(clf, X_test, y_test, n_repeats=10,                                random_state=42, n_jobs=2)sorted_idx = result.importances_mean.argsort()fig, ax = plt.subplots(figsize=(15,15))ax.boxplot(result.importances[sorted_idx].T,           vert=False, labels=X_test.columns[sorted_idx])ax.set_title("Permutation Importances (test set)")fig.tight_layout()plt.show()
From this we can see similar results to the feature importance results, showing essentially that free throws attempted and personal fouls add no information to the model.
Using this, we can create a decision rule to drop results from the model, which we will use if a feature contributes less than 1% feature importance. Which influence the results:
#extracting unimportant featuresUIF = list(fi[fi.importance <= 0.01].feature)X_trainIF = X_train.drop(columns = UIF)X_testIF = X_test.drop(columns = UIF)clf.fit(X_trainIF, y_train)test_pred_random_forest_IF = clf.predict(X_testIF)train_accuracy = clf.score(X_trainIF, y_train)test_accuracy = clf.score(X_testIF, y_test)oob_score = clf.oob_score_print(f"The training score of the random forest classifier is : {train_accuracy:.3f}")print(f"The out of basket score is: {oob_score:.3f}")print(f"The test score of the random forest classifier is {test_accuracy:.3f}")print (metrics.classification_report(y_test, test_pred_random_forest_IF))
Which, after dropping points, three point percentage, free throw percentage and free throws, the accuracy of the model has not changed overall! Thus reducing the complexity of the model while not changing the accuracy of the model. There is one more way of improving the model.
The final way we are going to attempt to improve the model is by tuning the hyperparameters, which are the parameters that influence how the model works and the random forest is created. For this we will select max_depth and min_samples_split which each influence how far each decision tree can go which would increase complexity, and how many samples are required before a branch can split. Increasing max depth increases complexity but could improve our model, and similarly with reducing the min samples split. Given that we want to be able to see whether we accurately predict NBA positions, our metric that we will train against, using grid search, is overall accuracy:
from sklearn.model_selection import GridSearchCVtuned_parameters = [{'max_depth': [4,5,6,7,8],                      'min_samples_split': [1,3,5,7,9]}]scores = ['accuracy']for score in scores:        print()    print(f"Tuning hyperparameters for {score}")    print()        clf = GridSearchCV(        RandomForestClassifier(), tuned_parameters,        scoring = f'{score}'    )    clf.fit(X_trainIF, y_train)        print("Best parameters set found on development set:")    print()    print(clf.best_params_)    print()    print("Grid scores on development set:")    means = clf.cv_results_["mean_test_score"]    stds = clf.cv_results_["std_test_score"]    for mean, std, params in zip(means, stds, clf.cv_results_['params']):        print(f"{mean:0.3f} (+/-{std*2:0.03f}) for {params}")
Which the results suggest that the best parameters for this is max_depth =7 and min_samples_split =9. Inputting these back into the model results in:
clf = RandomForestClassifier(n_estimators =100, oob_score=True, max_depth =6, min_samples_split = 7)clf.fit(X_trainIF, y_train)test_pred_random_forest = clf.predict(X_testIF)print (metrics.classification_report(y_test, test_pred_random_forest))
Which results in an overall increase in accuracy, but only a small one from 0.63 to 0.64.
Thus, using a Random Forest Classifier, an accuracy of only 0.64 suggests that using these statistics, we can predict position to a degree, however there may be some overlap in certain positions i.e. centre and power forward, point guard and shooting guard. This may suggest that there is a degree of positionlessness in the league, but the fact that the confusion matrix created in the first instance showed that no power forwards or centres were confused with point guards may suggest we are not in a truly positionless league where positions don’t matter at all.
This conclusion of course could also be influenced by the teams themselves, with the rise of small ball meaning that a traditional centre is not needed, such as Houston (Although they have issues of their own at the moment). There could also be more accurate statistics, such as advanced stats, that could be used to inform this and hence may suggest that each position may still be quite clearly defined, or indeed even other methods for classifying positions! Analysis could always be improved and in the future I hope to redo this analysis with different methods and potentially more stats, even comparing how the positions have changed over the years!
For full code see here