Interpreting Image Classification Model with LIME | by Ruben Winastwan | Towards Data Science

The advancement rate and growth in the area of machine learning are insane. Nowadays, we can choose a variety of machine learning models to solve our problems. Let’s say we want to solve a classification task, now we don’t only have logistic regression to choose from. We also have decision tree, random forest, SVM, Gradient Boosting, Neural Networks, and so on that will do the trick.
However, of all of the available machine learning models out there, we can agree that most of them are black boxes. This means that most machine learning models are doing something under the hood that is so complex that we don’t have a clue anymore why they behave the way they do. In other words, we don’t know the thought process of our model in predicting something.
Understanding the behavior of our machine learning model is becoming very important. Judging the model performance based on its accuracy alone is not sufficient anymore as your model can trick you. Let’s take a look at a ball classifier below which has one main job: to classify a ball either as a football or a basketball.
In the above image, our ball classifier performed really well. It predicted correctly the class of all of the six images. Next, we are happy with it and proceed to use this model for production. However, little did we know that our classifier has successfully tricked us. Why is that? Let’s take a look at the model explanation below.
It turns out that our classifier was correctly predicting a ball as a football because of the human body parts, not because of the ball itself. So our classifier wasn’t trying to classify between a football and a basketball, but to classify between a human body parts and a basketball. Obviously, it’s not what we want and thus, we shouldn’t trust our model only based on its accuracy.
Unfortunately, interpreting the behavior of our black-box model, like a deep neural network, is a very difficult thing to do.
And this is where we need LIME.
LIME stands for Local Interpretable Model-agnostic Explanations. It is a Python library based on a paper from Ribeiro et al. to help you understand the behavior of your black-box classifier model. Currently, you can use LIME for a classifier model that classify tabular data, images, or texts.
The abbreviation of LIME itself should give you an intuition about the core idea behind it. LIME is:
Model agnostic, which means that LIME is model-independent. In other words, LIME is able to explain any black-box classifier you can think of.
Interpretable, which means that LIME provides you a solution to understand why your model behaves the way it does.
Local, which means that LIME tries to find the explanation of your black-box model by approximating the local linear behavior of your model.
Let’s take a look at below picture to understand more about LIME being local.
Let’s say you have a feature space with non-linear boundaries as shown in the left image above. The data points residing inside the black curve will be classified as A and otherwise, will be classified as B. Now we want to predict the class of a data point denoted by red star in the above image. Instead of looking at the global behavior (left image), LIME will go into the vicinity area of the red star point such that it becomes very local that a linear classifier could explain your model’s prediction (right image).
Next, let’s take a look at how LIME explain the model’s behavior step-by step.
Internally, LIME tries to interpret a black box model by conducting these four steps:
1. Input data permutation
As you can see in the image above, let’s say we want LIME to explain why a data point denoted by the red star is classified in one of the class instead of the other. The first step that LIME would do is to create several artificial data points that are close with the data denoted by the red star.
If our input data is an image, LIME will generate several samples that are similar with our input image by turning on and off some of the super-pixels of the image.
2. Predict the class of each artificial data point
Next, LIME will predict the class of each of the artificial data point that has been generated using our trained model. If your input data is an image, then the prediction of each perturbed image will be generated at this stage.
3. Calculate the weight of each artificial data point
The third step is to calculate the weight of each artificial data to measure its importance. To do this, first the cosine distance metric is usually applied to calculate how far the distance of each artificial data point with respect to our original input data. Next, the distance will be mapped into a value between zero to one with a kernel function. The closer the distance, the closer the mapped value to one, and hence, the bigger the weight. The bigger the weight, the bigger the importance of a certain artificial data point.
If the input data is an image, then the cosine distance between each perturbed image and the original image will be computed. The more the similarity between a perturbed image to the original image, the bigger its weight and importance.
4. Fit a linear classifier to explain the most important features
The last step is fitting a linear regression model using the weighted artificial data points. After this step, we should get the fitted coefficient of each feature, just like the usual linear regression analysis. Now if we sort the coefficient, the features that have larger coefficients are the ones that play a big role in determining the prediction of our black-box machine learning model.
Now that we know how LIME works, let’s use it to explain the behavior of our machine learning model. In this article, we are going to create two image classification models: one with a custom model and one with a pre-trained InceptionV3 model.
Let’s load the data first.
The dataset that we’re going to use is the dog-cat-panda dataset that you can freely download on Kaggle. It contains in total 3,000 images of dogs, cats, and pandas.
If you’ve just downloaded it, then you need to unzip the file. Next, make sure that you have the following structure in your folders.
dog-cat-panda/        dog/      img_1.jpg      img_2.jpg      .....    cat/      img_1.jpg      img_2.jpg      .........    panda/      img_1.jpg      img_2.jpg      ...........
Now we can generate our training and test data by using image generator from TensorFlow. In the below code, we want to split the data into 80% of training data and 20% of test data.
And with that we have successfully generated our data!
Now it’s time for us to create our custom image classifier model. The model consists of three convolution layers before we flatten it and a fully-connected layer as the last layer of our model.
Next, let’s compile and train the model. For this article, we’re going to use Adam optimizer, accuracy metrics, and of course, categorical cross-entropy loss function.
Great! After this step now the model is ready to make a prediction of an input image. But before that, let’s create a pre-trained model based on InceptionV3. We can load the pre-trained model with one line of code as follows.
from tensorflow.keras.applications import inception_v3 as inc_net
Now let’s use the model to make a prediction of our input image and see how LIME could help us to understand the behavior of our model.
Let’s say we want our custom model to make a prediction of a panda image below:
The first step would be to preprocess our input image into the format that can be read by our custom model. After reading and transforming the input image, we can use our custom model to predict our input image.
And our custom model correctly predicts that the animal in the image is a panda! As you can see, our model predicts that with reasonable certainty, i.e with 81% probability.
But as mentioned earlier, we don’t know for sure why our model classifies the animal in the image as a panda. What makes our model think that the animal in our image is a panda instead of a dog or a cat? This is where we use LIME.
Before we start, if you haven’t installed LIME yet, you can do so by typing the following pip command.
pip install lime
Next, we need to import all of the necessary libraries. Since our input data is an image, we’re going to use LimeImageExplainer() method from lime_image . If your input data is a tabular data, you need to use LimeTabularExplainer() method from lime_tabular instead.
Now it’s time for us to start interpreting the prediction of our custom model. All we need to do is to call the explain_instance method from explainer object that we’ve created before.
As you can see above, we passed several arguments there:
images — The image that we want LIME to explain.
classifier_fn — Your image classier prediction function.
top_labels — The number of labels that you want LIME to show. If it’s 3, then it will only show the top 3 labels with highest probabilities and ignore the rest.
num_samples — to determine the amount of artificial data points similar to our input that will be generated by LIME.
Next, we can proceed to visualize the explanation provided by LIME.
Now we know why our model classifies our image as a panda! On the left image, we can see that only the super-pixels where the panda is visible are shown. This means that our model classifies our image as a panda because of these parts of super-pixels.
On the right image, the area of super-pixels colored in green are the ones that increase the probability of our image belongs to a panda class, while the super-pixels colored in red are the ones that decrease the probability.
Now we want to do the same step as above, but using the pre-trained InceptionV3 model. First, let’s see the prediction of the pre-trained InceptionV3 model using the same input image. Below is the code to do so.
As you can see, the pre-trained InceptionV3 model also predicts that our image is a panda. A giant panda to be precise. Now let’s interpret the behavior of our pre-trained model with the same step as our custom model before.
Now we also know why our pre-trained model classifies our image as a panda instead of a dog or a cat. It turns out that both of the custom model and pre-trained InceptionV3 model are able to classify our image as a panda due to the specific features of a panda, which is something that we want.
That’s it for now. Hopefully now you know the fundamental aspect of LIME and how it can be used to interpret our black-box machine learning model.
You can find the complete Jupyter Notebook covered in this article here.