LIME: Explaining predictions of machine learning models (1/2) | by Sharayu Rane | Towards Data Science

Feature Importance using Gini IndexFeature Importance using Permutation ImportancePartial Dependence Plots
Feature Importance using Gini Index
Feature Importance using Permutation Importance
Partial Dependence Plots
I would like to begin by asking the following question: “Can we trust the model predictions just because the model performance is convincingly high on the test data?” Many people might answer this question as ‘Yes’. But this is not always true. High model performance should not be considered an indicator to trust the model predictions, as the signals being picked up by the model can be random and might not make business sense.
In this blog I am going to talk about LIME , a method that helps in understanding the reasons behind predictions at an instance level. This is a model agnostic technique i.e. you can use this with any model be it Neural Networks, Tree based models, SVMs etc.
LIME stands for Local Interpretable Model-Agnostic Explanations. The word Local seems to be most confounding for many people. So let me start by explaining why the word Local.
The relation between the target and the independent variable can get very complex at a global level, for example refer to the image below. Looking at this global view does not provide much insight on the relationship between the independent feature and the target, so LIME zooms in goes at a very very local level.
Let’s assume that the instance being explained is marked by green star. The principal intuition behind LIME is that after zooming in and analyzing the relationship between target and the independent feature locally, we can approximate this relation using linear models.
Now let’s understand what are the steps of how LIME works:
Sample and SMOTE the instances in the original data close to the green star (instance being explained)
Calculate the distance between the sampled instances and the instance being explained
For these synthetically generated instances, make predictions using the original global model
Fit a simple linear model on this data set
This linear model is weighted based on the similarity index calculated in step 2. This is to ensure that the errors on the instances most closest to the original instance are valued more than others
Now let’s get into the example to explain the LIME results. I have used the data set from Kaggle. The objective is to identify the strongest predictors of default credit card payment. I have fitted a Scikit-learn implementation of Gradient Boosted Trees.
Correctly classified ‘Non Default Payment’ instance:
Orange color indicates that the contribution is towards a ‘Default Payment’, and Blue color indicates that the contribution is towards ‘Non Default Payment’ . The global model’s prediction for this instance is 0.072 where as for the local model the prediction is 0.066. So, for this particular instance the prediction of the global model and local model is very close.
The above example shows that the top contributor for this particular instance to be tagged as ‘Non Default Payment’ is the repayment status in the previous month (PAY_0 = 1). This means that in the previous month the payment was done on time. The interpretation of the other variables is also similar (PAY_2, PAY_3 etc.). This indicates that if the payments for previous months was done on time then there is high likelihood for the person to not default in the next month.
High amount of LIMIT_BAL also contributes towards ‘Non Default Payment’. This makes sense too because high LIMIT_BAL means that the supplementary credit left is low. So, there is a chance that the next month’s bill will potentially be low, and hence less chance to default. Now, for continuous features LIME’s output provides more detail as it specifies a range of feature values that are causing that feature to have its influence. E.g. ‘LIMIT_BAL > 240,000’
I personally find analyzing the incorrectly predicted instances to be very insightful. This helps in understanding what features actually drove the prediction to be incorrect.
Incorrectly classified ‘Non Default Payment’ instance (Actual: ‘Default Payment’):
The above instance is incorrectly predicted as Non Default Payment owing to the features that denote previous payment history (PAY_0, PAY_2, PAY_3). For this instance the previous payments were done duly on time. This might have swayed the prediction to be incorrect.
With the above example, the general idea of LIME sounds reasonable, but there are some potential drawbacks that LIME is susceptible to.
1. Local linear behavior
LIME currently implements linear models locally to get the explanations. While this assumption is valid for a small region around the instance being explained. But for more complex data sets, as we increase the area of the local region this assumption might not be true. Thus, the explanations from the local interpretable models might not explain the behavior of the global linear model.
2. Number of features
The number of features to be selected for the explanation needs to be optimally selected. The number of features should be selected such that the complexity of the model is maintained along with the simplicity of the explanation. In python implementation of LIME, it provides the following options: ‘forward_selection’, ‘lasso_path’, ‘none’ or ‘auto’.
3. Models without probability scores
LIME currently doesn’t support classifier models without probability scores.
4. No aggregated view
LIME implementation doesn’t provide an aggregated view of the feature explanation. The user has to in turn analyze the individual instances.
5. Similarity score
For tabular data, LIME creates new samples by perturbing each feature individually, drawing from a normal distribution with mean and standard deviation taken from the feature. For continuous features LIME provides an option to perturb samples from the normal centered on the instance being explained or from the normal centered on the mean of the feature data set. LIME weighs higher the samples close to the instance of interest. Now in high dimensional space with each feature having a different range of values, calculating similarity index is challenging and might not result in clean results.
6. Definition of the neighborhood
Defining a correct neighborhood to run the locally interpretable models is a difficult problem. So why do we need to define a local neighborhood along the instance being explained? The accuracy of the of the explanation model to the original model is enforced through the loss function over a set of samples in the simplified input space weighted by the local kernel. Currently LIME uses an exponential smoothing kernel. The width of this kernel determines the influence on the local model of the instance being explained, and again there is no best way to determine the width of the kernel.
7. Consistency of the model explanations
LIME explanations can sometimes lack consistency due to the sampling bias, calculation of the similarity score and definition of the neighborhood.
LIME provides very simple interpretable explanations and is a quick way to analyze the contribution of each feature. LIME can also be used on text and image data, and the execution time of LIME is less as compared to other explainable techniques such as SHAP. Another feature I like about LIME is that the local surrogate models can actually use independent features other than the ones used in the global model.
Additional reading resources:
https://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdfhttps://github.com/marcotcr/limehttps://christophm.github.io/interpretable-ml-book/lime.html
https://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf
https://github.com/marcotcr/lime
https://christophm.github.io/interpretable-ml-book/lime.html
In the next blog, I will explain another popular technique: SHAP that is used for explaining model predictions.
This field of Explainable AI is evolving rapidly, and there are lot of new developments in terms of tools and frameworks. In the comments section please write your feedback on the blog, and the latest tools you have used in this field. Also, comment if you would like me to write about any particular topic.
This content was originally published on my personal blogging website: http://datascienceninja.com/. Click here to view it and subscribe to receive updates on latest blogs instantly.