How to pick the RIGHT model. Looking at accuracy distribution, IQR... | by François St-Amant | Towards Data Science

Finding yourself with a few models/methods that perform almost equally is not rare in data science. When that happens, how do you know which model to choose? In this article, I will present a way to pick the best model, the real one.
I will be using the Breast Cancer Wisconsin Dataset, available here: https://www.kaggle.com/uciml/breast-cancer-wisconsin-data. It contains 569 observations and 32 descriptive variables. The dependent variable, diagnosis, is binary. It is equal to 1 if the cancer is malignant and to 0 if the cancer is benign. Let’s begin by importing the needed packages and the data.
library(ggplot2)library(randomForest)library (e1071)library(devtools)library(ggbiplot)#Import datadf = read.csv("data.csv")
Then, quickly, let’s clean the data (some columns are empty or useless) and transform the diagnosis variable into a [0,1] variable instead of a string.
#Y as factordf$diagnosis =  ifelse(df$diagnosis=="M", gsub("M", 1,                                                            df$diagnosis), gsub("B", 0, df$diagnosis))df$diagnosis = as.factor(df$diagnosis)#Remove useless columnsdf = df[,c(-33,-1)]
Here is the distribution of the dependent variable:
The classes don’t appear to be imbalanced. We can now move on to the next step. Below, I will create a “for loop”. In every iteration of the loop, I will separate data into two sets, training and validation. Afterwards, I will train and predict three separate models, using Random Forests, SVM and Logistic Regression. By putting all this into a “for loop” and changing the seed every time, it will allow me to each time collect the accuracy performance of the three models at each iteration. Changing the set.seed() changes the random repartition of data between train and validation.
This is essentially cross validation on three models at the same time. It is also beneficial because it allows me to compare the models on the same datasets every time. Each model is trained and validated on the same data, for every iteration. This means that we can make a fair and consistent comparison between each of the fitted models. Here is the code:
Here is the mean accuracy for the three models over the 200 iterations:
Random Forests: 96.7%SVM: 95.1%Logistic Regression: 97.1%
Random Forests: 96.7%
SVM: 95.1%
Logistic Regression: 97.1%
Logistic Regression seems to be the best algorithm for this dataset. However, to pick the RIGHT model, we cannot stop here.
Looking at the results above, we see that Logistic Regression wins but that the difference in its accuracy and the other models accuracy is not that significant. In order to know if we really have the best model, we need to look at the distribution of the accuracy measures. It is similar to the idea of marginal interpretation vs conditional interpretation: is the model good on average or is it always performing well, no matter the data.
This is useful because it lets us know if the models are always precise or if their performance is variable. We might want the best average model or the least variable model, depending on the situation. Having the accuracy distribution information can only help in making a coherent choice!
Here is how to get the plot of the distributions stored in the result_matrix, using the density function:
a=density(result_matrix[,1])#Getting the plotplot(a, xlim=c(0.92, 1), ylim=c(0, 60), col='blue', lwd=2, main='Accuracy distribution of all models',     xlab='Accuracy')lines(density(result_matrix[,2]), col='red', lwd=2)lines(density(result_matrix[,3]), col='green', lwd=2)legend(x=0.975, y=59,legend = c("Random Forest", "SVM", "Logistic Regression"), col = c("blue","red", 'green'), lty = c(1,1))
We see that Logistic Regression does seem to be narrowingly better than Random Forests, even if both accuracy distributions are spread pretty symmetrically and leanly around their mean (logistic regression seems to be ever so slightly leaner). Indeed, both distributions look normal. However, SVM distribution seems to be much more variable, with a much larger spread, and almost a second peek at around 95% accuracy.
We can also get box plots and the interquantile range, which will let us see if we have outliers (with box plots) and how narrow is the range between the 1st and 3rd quantile in our accuracy values (with IQR). Here is how to do it:
#Boxplotsboxplot(result_matrix, use.cols = TRUE,        main='Boxplot by algorithm',        ylab='Accuracy', xaxt = "n", col=c('blue', 'red', 'green'))axis(1, at=1:3, labels=c('Random Forests', 'SVM', 'Logistic           Regression'))#IQRIQR(result_matrix[,1])IQR(result_matrix[,2])IQR(result_matrix[,3])
Here are the IQRs for the three algorithms:
Random Forests: 1.05SVM: 1.4Logistic Regression: 1.05
Random Forests: 1.05
SVM: 1.4
Logistic Regression: 1.05
This means that for Random Forests and Logistic Regression, 50% of the values (between the first and the third quantile) are found in a range of 1.05% of accuracy. For SVM, the IQR is a bit wider, at 1.4%.
Looking at the boxplots, what we see is that for all three methods, we have very few outliers (1 or 2). Logistic Regression only has 1, while Random Forests has 2.
Now, I believe we can objectively say that the Logistic Regression algorithm is the best one to use in our case, and we have the data to prove it.
Thanks for reading!