GuidedGradCam on EfficientNet

CarlosHernandezP: <p>I’ve been working with the EfficientNet architecture lately and I wanted to apply Captum’s GuidedGradCam on it but I am getting strange results:</p>
<p>This is the code that I’m using:</p>
<pre><code class="lang-auto">from models.efficientnet import EfficientNet
import torch
import PIL.Image
from matplotlib import pyplot as plt

from torchvision.transforms import ToTensor, Resize, Compose, ToPILImage

from dataset_fun import get_data

device = 'cpu' ## Let's just put this to CPU for simplicity
model = EfficientNet.from_pretrained('efficientnet-b2', num_classes=2)
model.load_state_dict(torch.load(
    '$PATH_TO_MODEL/pre_trained_model.pt'))

df = get_data() # Getting my data
df = df[df['label'] == 1]

transforms = Compose([Resize((224, 224)), ToTensor()])

model = model.to(device)
model.eval()

# Here I Instantiate the grad cam class as well as the desired layer to compute the gradients
grad_cam_layer = model._conv_head # Last convolutional layer of the EfficientNet
guided_fc = attr.GuidedGradCam(model, grad_cam_layer)
for filename in df['filename'].values.tolist():
        # Some due pre-processing
        im = PIL.Image.open(filename)
        im = im.convert('RGB')
        im = transforms(im).unsqueeze(0).to(device)
        
        # Obtain attributes
        attributes = guided_gc.attribute(im, 1)

        # Plot them on screen
        plt.imshow(ToPILImage()(attributes[0])) # We convert to PIL and show it on screen
        plt.show()
        plt.clf()  # will make the plot window empty

</code></pre>
<p>Below I show the photo that was sent to the model and the returned GradCam (I know the photos aren’t pretty, skin cancer isn’t beautiful). As it can be seen, the result doesn’t provide much information.</p>
<p>Dissecting the EfficientNet I’ve seen that it doesn’t use any nn.ReLU but rather a novel non-linear function named Swish. Could this be causing the error as in the <a href="https://captum.ai/api/guided_grad_cam.html" rel="noopener nofollow ugc">Captum implementation</a> they mention that the ReLUs have to be of a specific type.</p>
<p>Any help on how to properly compute the Guided Gradient Cam of an image using Captum would be appreciated.</p>
<p>Caveat: I know the first image is bigger, but it gets resized in the transform.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93a6ea0d65bdcd0a1a7cf7a8ffd34345c0b7e92a.jpeg" alt="2021-02-05-130057_574x426_scrot" data-base62-sha1="l4bNCrwEo3QqTnHqJ6UmtS85Xqq" width="574" height="426"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73eea05e0497410eda5b2ca96741b190e26ff675.png" alt="2021-02-05-130206_441x417_scrot" data-base62-sha1="gxAa9XajdfcTCbp35R8TylMn8YB" width="441" height="417"></p>
<p>Advances:<br>
I managed to change the Swish to a nn.ReLU() with the following code:</p>
<pre><code class="lang-auto">for i in range(len(model._blocks)):
      model._blocks[i]._swish = nn.ReLU()
model._swish = nn.ReLU()
</code></pre>
<p>The problem is that it now outputs a complete black square.<br>
Althought this is the recommended activation function, the model wasn’t trained with it.</p>

CarlosHernandezP: <p><strong>Updates:</strong><br>
I retrained the model with nn.ReLU() instead of custom swish functions. The result is very similar.</p>
<p>Taking a look at this other <a href="https://discuss.pytorch.org/t/gradcam-zero-output/89496/2">post in the pytorch forum</a> one can see that <a href="https://freeimage.host/i/captum-results.dfkz0X" rel="noopener nofollow ugc">their result for GuidedGradCam</a> is more informative.</p>
<p>As it can be seen in my code, the in which I plot the image is via first converting it to a PIL image. Has anyone used Captum’s GuidedGradCam on a CNN that might be able to shed some light into why I’m having this output?</p>

ptrblck: <p>I’m not familiar with the underlying method in Captum, but based on your output image you might be running into clipping artifacts.<br>
I assume this line of code converts the wanted output to the <code>PIL.image</code>?</p>
<pre><code class="lang-python">ToPILImage()(attributes[0])
</code></pre>
<p>If so, could you check the stats of <code>attributes[0]</code> before converting it (min, max, mean, dtype etc.)?</p>

CarlosHernandezP: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>, thx for your answer:</p>
<p>Some info:<br>
attributes.shape <strong>[1,3,224,224]</strong> (that’s why I take the <strong>[0]</strong>)<br>
dtype: torch.float32<br>
min: -0.3880<br>
max: 0.2677<br>
mean: -3.287e-05 (isn’t it weird that it’s lower than the minimum?</p>
<p>It does seem that the values aren’t proper for an image visualization. Is it a preferred way to convert from tensor to image that does this under the hood?</p>

ptrblck: <p>Thanks for the information.</p>
<aside class="quote no-group" data-username="CarlosHernandezP" data-post="4" data-topic="111035">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carloshernandezp/48/48526_2.png" class="avatar"> CarlosHernandezP:</div>
<blockquote>
<p>mean: -3.287e-05 (isn’t it weird that it’s lower than the minimum?</p>
</blockquote>
</aside>
<p>The <code>mean</code> is not lower than the <code>min</code> (note the <code>e-05</code>), as it’s close to zero.</p>
<p>Based on the tensor stats I think you should normalize the tensor to have values in <code>[0, 1]</code>, as otherwise you’ll lose information.<br>
<code>ToPILImage</code> would call <a href="https://github.com/pytorch/vision/blob/ced96a0ca66e35ae61b64267e083fc3e3a62172a/torchvision/transforms/functional.py#L245-L246">these lines of code</a>:</p>
<pre data-code-wrap="python"><code class="lang-python">        if pic.is_floating_point() and mode != 'F':
            pic = pic.mul(255).byte()
</code></pre>
<p>which will add the clipping.<br>
Use:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(3, 224, 224)
x = x - x.min()
x = x / x.max()
</code></pre>
<p>to squash the values into <code>[0, 1]</code> before plotting them. You would still lose some information, but it might not be that disastrous.</p>

CarlosHernandezP: <p>Your snippet did the trick. Thank you for your time.</p>

ptrblck: <p>Cool! Out of curiosity: could you post the new output image?</p>

CarlosHernandezP: <p>The results are not very informative:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/2618710ca32a9710ccedc3e9e2a71170192d7694.png" alt="2021-02-08-123313_430x425_scrot" data-base62-sha1="5r0vBwsttRep5XiBz3xa5m25G1S" width="430" height="425"></p>
<p>But it does look like the classical GuidedGradCam <a href="https://haebinshin.github.io/public/img/grad-cam/guided_gradcam.png" rel="noopener nofollow ugc">as seen here</a>.</p>
<p>My next step now is to get the Grad Cam (similar thing but with a heatmap instead of a grayish image) which funnily enough that’s what I wanted on the first place.</p>