Uisng Lime Library with Pytorch on the function instance_explainer

katia_oussar: <p>Hello there :<br>
I am trying to use the lime.lime_image.LimeImageExplainer().explain_instance function to interpret the results of my trained from scratch Swin Transformer on 100 000  galaxy images. I am not quite sure how the classifier_fn works. the function takes in two required parameters: the image which should be a numpy array and a th classifier_fn which takes that same image, converts it to a tensor and produces the probabilities. The following is snippet of my implemented code and I’d very much appreciate the help on making it work as the issue is high priority and requires immediate attention.<br>
Thank you : )</p>
<h3><a name="h-1-create-a-swin-transformer-model-instance-1" class="anchor" href="#h-1-create-a-swin-transformer-model-instance-1"></a>1- Create a Swin Transformer Model Instance</h3>
<h3><a name="h-2-load-the-weights-to-the-swin-instance-2" class="anchor" href="#h-2-load-the-weights-to-the-swin-instance-2"></a>2- Load the weights to the Swin Instance</h3>
<h3><a name="h-3-read-teh-image-and-apply-transformations-to-it-3" class="anchor" href="#h-3-read-teh-image-and-apply-transformations-to-it-3"></a>3- Read  teh image and apply transformations to it</h3>
<h3><a name="h-4-apply-xai-4" class="anchor" href="#h-4-apply-xai-4"></a>4- Apply XAI.</h3>
<p>import torch<br>
import numpy as np<br>
import torch.nn.functional as F<br>
from lime import lime_image<br>
from torchvision import transforms<br>
from PIL import Image<br>
from skimage.segmentation import mark_boundaries<br>
import matplotlib.pyplot as plt<br>
from swin_transformer_pytorch.swin_transformer import SwinTransformer</p>
<h3><a name="h-1-model-instantiation-5" class="anchor" href="#h-1-model-instantiation-5"></a>1 : Model Instantiation</h3>
<p>model = SwinTransformer(hidden_dim = 96, layers=(2,2,6,2), heads= (3, 6, 12, 24), channels=3, num_classes= 8, head_dim=32, window_size=7,<br>
downscaling_factors=(4, 2, 2, 2), relative_pos_embedding=True)</p>
<h3><a name="h-2-loading-the-model-weights-6" class="anchor" href="#h-2-loading-the-model-weights-6"></a>2 : Loading the model weights</h3>
<p>dictionary_path = “C:\Users\katy99\Desktop\MyCiApp\gz2_datasets\Swin_7_2023_cache.pth”</p>
<p>model.load_state_dict(torch.load(dictionary_path, map_location=torch.device(‘cpu’)))<br>
model.eval()</p>
<h3><a name="h-3-preprocessing-the-test-iamge-7" class="anchor" href="#h-3-preprocessing-the-test-iamge-7"></a>3 : Preprocessing the test iamge:</h3>
<p>image_path = ‘C:\Users\katy99\Desktop\MyCiApp\images_test\49.jpg’<br>
input_size = 224<br>
test_transform = transforms.Compose([transforms.CenterCrop(input_size),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.094, 0.0815, 0.063], [0.1303, 0.11, 0.0913])])</p>
<p>image = Image.open(image_path)<br>
input_image = test_transform(image).unsqueeze(0)<br>
print (input_image.shape)</p>
<p><span class="hashtag-raw">#Step</span> 4: Use LIME to generate the XAI:</p>
<h1><a name="crucial1-reshape-the-image-tensor-to-a-nunpy-array-for-the-first-parameter-8" class="anchor" href="#crucial1-reshape-the-image-tensor-to-a-nunpy-array-for-the-first-parameter-8"></a>Crucial1: Reshape the  image tensor to a nunpy array for the first parameter,</h1>
<h1><a name="which-is-the-input-image-9" class="anchor" href="#which-is-the-input-image-9"></a>Which is the input image</h1>
<p><span class="hashtag-raw">#Crucial2</span> : Reshape the array back to a tensor for the forward function</p>
<h1><a name="requires-the-writing-of-my-own-function-10" class="anchor" href="#requires-the-writing-of-my-own-function-10"></a>Requires the writing of my own function.</h1>
<p>input_image_numpy = input_image[0].permute(1,2,0).numpy()</p>
<p>def classifier_fn (numpy_image):<br>
torch_image = torch.from_numpy(numpy_image)<br>
with torch.no_grad():<br>
outputs = model (torch_image)<br>
_, predicted_class = torch.max(outputs, 1)<br>
probs = F.softmax(predicted_class, dim=1)<br>
return probs.detach().cpu().numpy()</p>
<p>explainer = lime_image.LimeImageExplainer()<br>
explanation = explainer.explain_instance(input_image_numpy, classifier_fn, top_labels=5, num_samples=1000)</p>
<p>temp, mask = explanation.get_image_and_mask(explanation.top_labels[0],<br>
positive_only=True,<br>
num_features=5,<br>
hide_rest=False)</p>
<h1><a name="step-5-visualize-the-explanations-11" class="anchor" href="#step-5-visualize-the-explanations-11"></a>Step 5: Visualize the explanations</h1>
<h1><a name="plot-the-original-image-with-highlighted-important-regions-12" class="anchor" href="#plot-the-original-image-with-highlighted-important-regions-12"></a>Plot the original image with highlighted important regions</h1>
<p>plt.imshow(mark_boundaries(temp / 2 + 0.5, mask))<br>
plt.axis(‘off’)<br>
plt.show()</p>