Machine Learning Model Implementation: Assessing Variable Importance Across Models | by Katey Durham | Towards Data Science

Great! So you completed your Machine Learning (ML) pipeline, hopefully choosing your tuning parameters for each model through a careful validation procedure and have converged on what you think is the final model for each strategy. What’s next? In the first part of this series I’ll talk about some metrics and graphics beyond the area under the ROC curve that can be helpful in achieving our project’s goals. Sometimes as data scientists our tunnel vision for the data and modeling, or at least others’ perception of us in this way, can limit our potential effectiveness as communicators and strategizers. The elements I’ll demonstrate in these articles will help you see the problems you work on through your collaborator’s eyes and provide concrete ways for you to address their concerns and objectives for your ML work. Specifically, in this article we’ll:
Discuss why a unifying framework to explore Variable Importance in Machine Learning is, well, important!Display R code to produce a Variable Importance plot using the results from a group of ML models.
Discuss why a unifying framework to explore Variable Importance in Machine Learning is, well, important!
Display R code to produce a Variable Importance plot using the results from a group of ML models.
Hopefully, you are sold already on the importance of trying different approaches when you tackle a prediction problem. Fitting models of varying complexity (e.g. both linear and non-linear) can identify the most parsimonious model that gives good prediction performance for the data we’re working with. However, the flip side to the advantages that come with applying different models can be the apparent difficulty in comparing the results across these divergent strategies. What about the predictors that were used to fit the models — how can these be evaluated to determine what was important? As a data scientist, I challenge you to find a subject matter expert who, although you may deliver a model that predicts quite well, has no curiosity about what is happening under the hood! Your final model may have a straightforward interpretation of the parameter coefficients, such as for a linear model, but this is not always the case for more complex non-linear strategies. Luckily, although models have different parameterization and optimization strategies, Relative Variable Importance is a unifying concept that can allow comparisons to be made concerning features across models. More details on how Importance is computed for different models can be found in Applied Predictive Modeling (http://appliedpredictivemodeling.com) by Max Kuhn and Kjell Johnson, and can be implemented in the caret package in R.
Basically the unification is achieved in this way: although a variable’s Importance is determined differently for each model, the single most important variable for every strategy is given a score of 100, as in ‘100 percent,’ so that it’s most accurate to describe these importance scores as relative. Then the next important variable is assigned a relative value between 0 and 100, where its proximity to 100 indicates how close in importance it is to the first variable. This proceeds throughout the predictors, except in the case of a relatively small number of models that perform their own variable selection through the model fitting procedure. For these models, the variable importance is zero for any feature not included in the model.
One important caveat to note about Relative Variable Importance is that it’s best thought of as an indicator of absolute value, but not sign or direction, of the effect. Values of important predictors may be either positively or negatively associated with the outcome.
In this article, I’ll be using results from 7 ML models used to analyze the German Credit classification data available in the R caret package. It consists of 1000 observations of individuals determined to have either Good (n=700) or Bad (n=300) credit. There are 61 predictors that cover a variety of factors that may have to do with credit, such as loan features, banking information, demographics, employment, etc. The data were separated into a 70% — 30% training — test split, preserving the 70% Good credit frequency in both. Note that computing the variable importance only involves data from the training set, which was split into 5 CV sets, repeated 5 times, to determine optimal tuning parameters. Modeling was done using the caret package in R. The models fit were the following:
* Linear Discriminant Analysis (LDA)* Partial Least Squares (PLS)* Support Vector Machines (SVM, with radial kernel function)* Neural Network (NN)* Recursive Partitioning (Rpart, single tree) * Random Forests (RF)* Gradient Boosting Machines (GBM)
In the next article in the series, I’ll talk more about how these models performed. For our purpose now the models indicated a reasonable signal in the cross-validation performance so it is warranted to look at the Relative Variable Importance.
In R, variable importance measures can be extracted from caret model objects using the varImp() function. Here, though, we’ll pick things up in the code from a .csv file containing the top 10 important variables from each model, along with their Importance value, so you can join the code here in R if you have a file like this from another source.
require(tidyverse)require(ggplot2)require(ggrepel)varImp_all <- read_csv('varImp.csv')varImp_all## # A tibble: 70 x 3##    model imp_vars                   Importance##    <chr> <chr>                           <dbl>##  1 lda   CheckingAccountStatus.lt.0      100  ##  2 lda   CheckingAccountStatus.none       87.8##  3 lda   InstallmentRatePercentage        54.3##  4 lda   Duration                         49.9##  5 lda   CreditHistory.Critical           47.4##  6 lda   Telephone                        42.2##  7 lda   SavingsAccountBonds.lt.100       38.7##  8 lda   OtherInstallmentPlans.None       38.3##  9 lda   Age                              36.5## 10 lda   Housing.Own                      34.8## # ... with 60 more rows
In the next steps the data is processed to achieve the 3 dimensions captured on the plot:
The number of models in which the feature appeared in the top 10The feature’s average ranking across the modelsThe feature’s average Relative Importance value across the models
The number of models in which the feature appeared in the top 10
The feature’s average ranking across the models
The feature’s average Relative Importance value across the models
# Variable importance figure# create rankings for important variables in each modelvarImp_all$rank <- sequence(rle(as.character(varImp_all$model))$lengths)# compute average importance and ranking for all variablesvarImpSummary <- aggregate(varImp_all[,3:4], list(varImp_all$imp_vars), mean)colnames(varImpSummary) <- c('var','avg_imp','avg_rank')varImpSummary <- varImpSummary %>%  mutate(avg_imp = round(avg_imp,2),         avg_rank = round(avg_rank,2))# count frequency of important variable across modelsvarImpTable <- data.frame(table(varImp_all$imp_vars))varImpOrder <- varImpTable[order(varImpTable$Freq, decreasing = T),]colnames(varImpOrder) <- c("var", "num_models")varImp_dat <- merge(varImpSummary,varImpOrder)varImp_dat <- varImp_dat[order(varImp_dat$avg_imp, decreasing = TRUE),]
Below is the ggplot2 code. Probably the trickiest part is to get both size and color in a single legend. Also, we are reversing the usual direction of the y-axis, so that that a ranking of one will be on top. This will create the sweet spot on the graph at the upper right corner, where the features with the highest rankings across multiple models will congregate. Similarly the lower left will represent the least important variables.
p <- ggplot(varImp_dat, aes(x = num_models, y = avg_rank)) +   geom_point(aes(color = avg_imp, size = avg_imp)) +   scale_size_continuous(range = c(1,30)) +  scale_color_gradient(low = "yellow", high = "hotpink", na.value = NA, guide = 'legend') +  guides(color=guide_legend(reverse = TRUE), size = guide_legend(reverse = TRUE)) +  scale_y_continuous(trans = "reverse") +   labs(title = "Variable Importance Across Models",        x = "Number of Models Where Variable Appeared in Top 10",        y = "Average Ranking", color = "Average\nImportance", size = "Average\nImportance")
This section of the plot code labels the predictors. The geom_label_repel() function from the ggrepel package can help position labels strategically (i.e. not on top of each other).
p <- p + geom_label_repel(aes(label = var),                          box.padding   = 0.35,                           point.padding = 0.5,                          segment.size = 0.5,                          size = 6,                          segment.color = 'grey50')
And lastly some theme adjustments:
p <- p +   theme(plot.title = element_text(color="black",face="bold",size=18,hjust=0),        axis.title = element_text(color="black",face="bold",size=12),        legend.title = element_text(color="black",face="bold",size=12),        legend.position = "right",        axis.text=element_text(size=12))
Finally, we have a visual summary of the important variables as indicated by the models in this analysis. Probably not surprisingly, having a checking account with a balance less than zero, or not having a checking account at all were important factors across multiple models. Also, features of the loan itself, including the installment rate and duration, appeared to be important in all models. Less frequently appearing but important in the models in which it was found was the amount of the loan. Only one model (it happened to be the Neural Net) picked up on the purpose of the loan as providing furniture and equipment, but it displayed a relatively high importance value. Perhaps the Neural Net is picking up on loans for business, such as restaurants, with a relatively high propensity to fail. These types of insights have the potential to perhaps streamline data collection and suggest ways to improve the loan approval process to decrease defaults.
In conclusion, if the purpose of your modeling is to identify ways to improve performance in your process, not merely to monitor and predict, having some insight into the drivers of your outcome will be highly valuable. Being able to display the Relative Variable Importance will likely pique interest in the model’s utility and allow for more strategic implementation.