Decision Trees — Pruning. My last blog focused on the concept of... | by Blake Lawrence | Towards Data Science

My last blog focused on the concept of decision trees which form the basis of the Random Forest machine learning algorithm. As it was only a short blog (4 minute read), I didn’t delve into the detail however did embed some (hopefully) useful links.
In this post I would like to go a little further and cover:
How random forest uses decision trees
The problem of over-fitting and how you can potentially identify it
Pruning decision trees to limit over-fitting issues.
As you will see, machine learning in R can be incredibly simple, often only requiring a few lines of code to get a model running. Although useful, the default settings used by the algorithms are rarely ideal.
The following code is an example to prepare a classification tree model. I have used the ‘rpart’ package however ‘caret’ is another alternative.
Model = rpart(classifier ~ predictor, data = train_data, method = “class”, control = rpart.control(###))
The ‘classifier’ is the feature to be predicted by the model, and the ‘predictors’ are the independent features within the dataset which will be used to determine probabilities for the classifier outcome.
The method = “class” statement will ensure a prediction is made for the classifier (for a factor type variable). For example, preparing scripts relating to a loan repayment data-set. The classifier was whether an applicant had ‘repaid’ or ‘defaulted’ on a debt with a number of features (columns) available to predict this outcome. The random forest algorithm uses a number of decision trees prepared using random segments of the training data. Each of the observations in the test data set is then run through each of the trees to independently make predictions as to the outcome of the classifier. The classification predicted by the largest number of trees represents the prediction of the model. If most of the trees predict a loan will be repaid, than this is the prediction of the model.
Test_data$Prediction <- predict(Model, data = test_Data, type = “class”)
Once a prediction vector was created, the overall accuracy of the models could be calculated and a confusion matrix produced showing which of the two outcomes (repaid and default) the model was best at predicting.
Accuracy = mean(test_data$Predicted == test_data$actual_outcome)
Confusion Matrix = table(test_data$Predicted,test_Data$actual_outcome)
In one of the activities I completed in Data Camp, I went a little further than the exercise required to test the accuracy of both the standard decision tree as well as a version where I had pruned the tree. The reason for pruning is that the trees prepared by the base algorithm can be prone to overfitting as they become incredibly large and complex. Below I have shown the decision tree produced using the base algorithm with no tuning parameters. Don’t worry that the font is too small to read, I am simply highlighting just how complex the tree can be with no control parameters added.
When I pruned the tree I tested different values for minsplit (number of observations to appear in a leaf before it is ‘pruned’) and maxdepth (maximum number of nodes in a branch). This terminology is all covered in my earlier blog if you would appreciate a quick refresh!
I found that a tree of significantly reduced complexity actually provided a small increase in accuracy — I was aiming to keep accuracy stable! The original model produced similar levels of accuracy across both default and repaid home loans whereas the new model has improved significantly at predicting default loans with a smaller deterioration in accuracy on its repaid loan predictions. This would be worth exploring more however the drastic reduction in complexity has still yielded excellent results.
This is the code I prepared for both the standard and pruned trees. Only a small amount of code with lots of room for further improvement!
# Trees prepared with no pruning# prepare modellm <- rpart(outcome ~ ., data = loans_train, method = “class”, control = rpart.control(cp = 0))# prepare predictionloans_test$PredLM <- predict(lm,loans_test, type = “class”)# calculate overall accuracymean(loans_test$PredLM == loans_test$outcome)# prepare confusion matrixtable(loans_test$PredLM,loans_test$outcome)# plot treerpart.plot(lm, type = 3, box.palette = c(“red”, “green”), fallen.leaves = TRUE)# Trees prepared with pruning# prepare modellmp <- rpart(outcome ~ ., data = loans_train, method = “class”, control = rpart.control(cp = 0, minsplit = 250, maxdepth = 6))# prepare predictionloans_test$PredLMP <- predict(lmp,loans_test, type = “class”)# calculate overall accuracymean(loans_test$PredLMP == loans_test$outcome)# prepare confusion matrixtable(loans_test$PredLMP,loans_test$outcome)# plot treerpart.plot(lmp, type = 3,cex = .7, box.palette = c(“red”, “green”), fallen.leaves = TRUE)