Understanding Deep Learning Models with Integrated Gradients | by Renu Khandelwal | Towards Data Science

This post will help you to understand the two basic axioms of Integrated Gradients and how to implement Integrated Gradient using TensorFlow using a transfer learned model.
What is Integrated Gradient?
Integrated Gradient(IG) is an interpretability or explainability technique for deep neural networks which visualizes its input feature importance that contributes to the model's prediction
Can IG be applied to only a specific use case of deep learning or only to a specific neural network architecture?
Integrated Gradient(IG) computes the gradient of the model’s prediction output to its input features and requires no modification to the original deep neural network.
IG can be applied to any differentiable model like image, text, or structured data.
IG can be used for
Understanding feature importance by extracting rules from the network
Debugging deep learning models performance
Identifying data skew by understanding the important features contributing to the prediction
How does Integrated Gradient work?
Explaining IG using a deep learning model for image classification
Integrated Gradient is built on two axioms which need to be satisfied:
Sensitivity andImplementation Invariance
Sensitivity and
Implementation Invariance
To calculate the Sensitivity, we establish a Baseline image as a starting point. We then build a sequence of images which we interpolate from a baseline image to the actual image to calculate the integrated gradients.
Implementation invariance is satisfied when two functionally equivalent networks have identical attributions for the same input image and the baseline image.
Two networks are functionally equivalent when their outputs are equal for all inputs despite having very different implementations.
Step 1: Start from the baseline where baseline can be a black image whose pixel values are all zero or an all-white image, or a random image. Baseline input is one where the prediction is neutral and is central to any explanation method and visualizing pixel feature importances.
Step 2: Generate a linear interpolation between the baseline and the original image. Interpolated images are small steps(α) in the feature space between your baseline and input image and consistently increases with each interpolated image’s intensity.
Step 3: Calculate gradients to measure the relationship between changes to a feature and changes in the model’s predictions.
The gradient informs which pixel has the strongest effect on the models predicted class probabilities.
Varying variable changes the output, and the variable will receive some attribution to help calculate the feature importances for the input image. A variable that does not affect the output gets no attribution.
Step 4: Compute the numerical approximation through averaging gradients
Step 5: Scale IG to the input image to ensure that the attribution values are accumulated across multiple interpolated images are all in the same units. Represent the IG on the input image with the pixel importances.
How to implement an Integrated Gradient using Tensorflow?
Importing required libraries
import matplotlib.pylab as pltimport numpy as npimport tensorflow as tfimport tensorflow_hub as hubfrom tensorflow.keras.applications.mobilenet_v2 import preprocess_input as mobilenet_v2_preprocess_input
Using MobileNetV2 as Transfer learned model on Imagenet dataset
model = tf.keras.applications.MobileNetV2(input_shape=(224,224,3),                                               include_top=True,                                               weights='imagenet')
Loading Imagenet labels
def load_imagenet_labels(file_path):  labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path)  with open(labels_file) as reader:    f = reader.read()    labels = f.splitlines()  return np.array(labels)imagenet_labels = load_imagenet_labels('https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt')
Load and preprocess images
def read_image(file_name):  image = tf.io.read_file(file_name)  image = tf.image.decode_jpeg(image, channels=3)  image = tf.image.convert_image_dtype(image, tf.float32)    image = tf.image.resize_with_pad(image, target_height=224, target_width=224)  return imageimg = {'Peacock':'Peacock.jpg'}img_name_tensors = {name: read_image(img_path) for (name, img_path) in img.items()}
Display the original input image
plt.figure(figsize=(5, 5))ax = plt.subplot(1, 1, 1)ax.imshow(img_name_tensors['Peacock'])ax.set_title("Image")ax.axis('off')plt.tight_layout()
Predict the top three predictions from the model for the input image
def top_k_predictions(img, k=3):  image = tf.expand_dims(img, 0)  predictions = model(image)  probs = tf.nn.softmax(predictions, axis=-1)  top_probs, top_idxs = tf.math.top_k(input=probs, k=k)  top_labels = np.array(tuple(top_idxs[0]) )  return top_labels, top_probs[0]#Display the image with top 3 prediction from the modelplt.imshow(img_name_tensors['Peacock'])plt.title(name, fontweight='bold')plt.axis('off')plt.show()pred_label, pred_prob = top_k_predictions(img_name_tensors['Peacock'])for label, prob in zip(pred_label, pred_prob):    print(f'{imagenet_labels[label+1]}: {prob:0.1%}')
Create a black baseline image which will act as a starting point for calculating feature importance
baseline = tf.zeros(shape=(224,224,3))
Generate linear interpolation between baseline and the original input image
m_steps=50alphas = tf.linspace(start=0.0, stop=1.0, num=m_steps+1) def interpolate_images(baseline,                       image,                       alphas):  alphas_x = alphas[:, tf.newaxis, tf.newaxis, tf.newaxis]  baseline_x = tf.expand_dims(baseline, axis=0)  input_x = tf.expand_dims(image, axis=0)  delta = input_x - baseline_x  images = baseline_x +  alphas_x * delta  return imagesinterpolated_images = interpolate_images(    baseline=baseline,    image=img_name_tensors['Peacock'],    alphas=alphas)
Visualizing the interpolated image
fig = plt.figure(figsize=(20, 20))i = 0for alpha, image in zip(alphas[0::10], interpolated_images[0::10]):  i += 1  plt.subplot(1, len(alphas[0::10]), i)  plt.title(f'alpha: {alpha:.1f}')  plt.imshow(image)  plt.axis('off')plt.tight_layout();
Compute gradients between model outputs and interpolated inputs
Computing gradients measures the relationship between changes to a feature and changes in the model’s predictions. We use tf.GradientTape to compute the gradients between the interpolated image along with the top predicted class Id indicating which pixels have the strongest influence on the models’ prediction
def compute_gradients(images, target_class_idx):  with tf.GradientTape() as tape:    tape.watch(images)    logits = model(images)    probs = tf.nn.softmax(logits, axis=-1)[:, target_class_idx]  return tape.gradient(probs, images)path_gradients = compute_gradients(    images=interpolated_images,    target_class_idx=84)
Accumulate the Gradients using the Riemann Trapezoid
def integral_approximation(gradients):  # riemann_trapezoidal  grads = (gradients[:-1] + gradients[1:]) / tf.constant(2.0)  integrated_gradients = tf.math.reduce_mean(grads, axis=0)  return integrated_gradients
Putting all the steps in a function to calculate the integrated gradients
@tf.functiondef integrated_gradients(baseline,                         image,                         target_class_idx,                         m_steps=50,                         batch_size=1):  # 1. Generate alphas.  alphas = tf.linspace(start=0.0, stop=1.0, num=m_steps+1)# Initialize TensorArray outside loop to collect gradients.      gradient_batches = tf.TensorArray(tf.float32, size=m_steps+1)      # Iterate alphas range and batch computation for speed, memory #efficiency, and scaling to larger m_steps.  for alpha in tf.range(0, len(alphas), batch_size):    from_ = alpha    to = tf.minimum(from_ + batch_size, len(alphas))    alpha_batch = alphas[from_:to]# 2. Generate interpolated inputs between baseline and input.    interpolated_path_input_batch = interpolate_images(baseline=baseline,                                                       image=image,                                                       alphas=alpha_batch)# 3. Compute gradients between model outputs and interpolated inputs.    gradient_batch = compute_gradients(images=interpolated_path_input_batch,                                       target_class_idx=target_class_idx)        # Write batch indices and gradients to extend TensorArray.    gradient_batches = gradient_batches.scatter(tf.range(from_, to), gradient_batch)        # Stack path gradients together row-wise into single tensor.  total_gradients = gradient_batches.stack()# 4. Integral approximation through averaging gradients.  avg_gradients = integral_approximation(gradients=total_gradients)# 5. Scale integrated gradients with respect to input.  integrated_gradients = (image - baseline) * avg_gradientsreturn integrated_gradientsig_attributions = integrated_gradients(baseline=baseline,                                       image=img_name_tensors['Peacock'],                                       target_class_idx=84,                                       m_steps=283)
Visualizing the attribution and the Integrated gradient to explain the prediction for the input image
def plot_img_IG(baseline,                          image,                          target_class_idx,                          m_steps=50,                          cmap=None,                          overlay_alpha=0.4):  attributions = integrated_gradients(baseline=baseline,                                          image=image,                                      target_class_idx=target_class_idx,                                      m_steps=m_steps)  attribution_mask = tf.reduce_sum(tf.math.abs(attributions), axis=-1)  fig, axs = plt.subplots(nrows=1, ncols=2, squeeze=False, figsize=   (8, 8))  axs[0, 0].set_title('Attribution mask')  axs[0, 0].imshow(attribution_mask, cmap=cmap)  axs[0, 0].axis('off')  axs[0, 1].set_title('Overlay IG on Input image ')  axs[0, 1].imshow(attribution_mask, cmap=cmap)  axs[0, 1].imshow(image, alpha=overlay_alpha)  axs[0, 1].axis('off')  plt.tight_layout()  return fig_ = plot_img_IG(image=img_name_tensors['Peacock'],                          baseline=baseline,                          target_class_idx=84,                          m_steps=240,                          cmap=plt.cm.inferno,                          overlay_alpha=0.4)
Integrated Gradient(IG) helps you explain what a deep learning model looks at to make a prediction by highlighting the feature importances. It does this by computing the gradient of the model’s prediction output to its input features. It does not need any modification to the original deep neural network and can be applied to images, text as well as structured data. IG is based on two axioms of Sensitivity and Implementation Invariance.
Axiomatic Attribution for Deep Networks