Feature visualization not repeateable using torchcam for a CNN classifier

Linus: <p>Hello PyTorch forums, and thanks for all the help you have provided me so far!</p>
<p>I’m trying to visualize the features (filters) of my Resnet CNN when applied to a binary classification problem. The goal is to see somehow how my model is interpreting images of sawn timber when classifying them as either A or B. Ideally I would like to see a feature map highlighting (big weights) things like knots as the deciding factor for the classification.</p>
<p>My current attempt is using the <a href="https://github.com/frgfm/torch-cam" rel="noopener nofollow ugc">torch-cam</a> package from github, which to my understanding is the go-to method, yes? My code with some comments/thoughts is:</p>
<pre><code class="lang-auto">!pip install torchcam
from torchcam.cams import SmoothGradCAMpp

# model.eval() is exectured prior to the code in this post.
model.to('cpu')

# Using a fixed sample image
img = Image.open('/content/drive/My Drive/data/Lundgrens collection/A/63546298.png')

# Transformations are resize, to tensor, and normalize.
# Shape becomes [1, 3, 50, 783] which is very wide!
img_tensor = torch.unsqueeze(transformations(img),0)

# The image colors look very strange, I think this is due to the normalization, which is ok I guess.
fig, ax = plt.subplots(figsize=(36, 6))
ax.imshow(np.transpose(img_tensor[0], (1, 2, 0)), interpolation='nearest')

### Layer 1
# Hook your model before the forward pass
cam_extractor = SmoothGradCAMpp(model, 'layer1') # What is "layer1" here? See below.
# By default the last conv layer will be selected
out = model(img_tensor)
# Retrieve the CAM
activation_map = cam_extractor(out.squeeze(0).argmax().item(), out)

fig, ax = plt.subplots(figsize=(36, 6))
ax.imshow(activation_map, interpolation='nearest')
</code></pre>
<p>The problem is that if I run this code repeatedly the feature visualization is never the same. It is completely different, seemingly as if a random image is being sent to the model. Furthermore, I don’t think I understand the hooking of “layer1”, since as shown below layer1 consists of several sub-layers. So maybe I just don’t understand how to use the torch-cam package? Reading the source code is a bit too complicated to me and I don’t understand it enough to tell what is wrong here.</p>
<pre><code class="lang-auto">conv1.weight
bn1.weight
bn1.bias
layer1.0.conv1.weight
layer1.0.bn1.weight
layer1.0.bn1.bias
layer1.0.conv2.weight
layer1.0.bn2.weight
layer1.0.bn2.bias
layer1.1.conv1.weight
layer1.1.bn1.weight
layer1.1.bn1.bias
layer1.1.conv2.weight
layer1.1.bn2.weight
layer1.1.bn2.bias
layer2.0.conv1.weight
...
</code></pre>
<p>Thanks for any help or feedback you could provide! I’m no expert on pytorch so don’t hold back if I’m doing something silly.</p>

thomasjo: <aside class="quote no-group quote-modified" data-username="Linus" data-post="1" data-topic="116601">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/f4b2a3/48.png" class="avatar"> Linus:</div>
<blockquote>
<pre><code class="lang-auto"># Model is in eval() mode.
model.to('cpu')
</code></pre>
</blockquote>
</aside>
<p>I’m a little bit confused by this comment since that line of code doesn’t set the model in evaluation mode, it only moves the model from the source device to the <code>"cpu"</code> device.</p>
<p>But maybe your comment means that somewhere else (in code you’ve not shared) you’ve called <code>model.eval()</code>? Just want to make sure, because judging from your description it definitely seems like the model is <strong>not</strong> in evaluation mode.</p>

Linus: <p>You are correct, That code is run previously, I’ll make an edit to try and make it more clear, thanks!</p>

thomasjo: <p>OK, so I looked at the source code for <code>SmoothGradCAMpp</code>, and they’re adding random noise to the input, which explains what you’re describing. If you’re interested, you can find the relevant line of code here: <a href="https://github.com/frgfm/torch-cam/blob/b1c61003f036b754cc3bf21fc24208da96f25019/torchcam/cams/gradcam.py#L258-L259" class="inline-onebox" rel="noopener nofollow ugc">torch-cam/gradcam.py at b1c61003f036b754cc3bf21fc24208da96f25019 · frgfm/torch-cam · GitHub</a></p>
<p>If you want to get the same output for every subsequent run of your code, the simplest solution is to use a fixed seed for PyTorch’s random number generator;</p>
<pre><code class="lang-auto">torch.random.set_manual_seed(42)  # Change '42' to whatever you want
</code></pre>
<p>Hope this helps.</p>

Linus: <p>Thanks that really explains things!</p>
<p>Why though?? Isn’t the point of feature visualization to understand how the weights behave for an image? In my case, it seems that the noise is dominating and more or less the only thing I see is something related to this noise. The feature maps are always completely unrelated to the previous iteration.</p>
<p>Am I misunderstanding something here, why would you want this? Can I turn the noise completely off?</p>
<p>Edit: I remembered something and found out that I already do this!</p>
<pre><code class="lang-auto">torch.manual_seed(42)
</code></pre>

thomasjo: <aside class="quote no-group" data-username="Linus" data-post="5" data-topic="116601">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/f4b2a3/48.png" class="avatar"> Linus:</div>
<blockquote>
<p>Why though?? Isn’t the point of feature visualization to understand how the weights behave for an image?</p>
</blockquote>
</aside>
<p>The reasoning can be found in the <a href="https://arxiv.org/pdf/1908.01224.pdf" rel="noopener nofollow ugc">Smooth Grad-CAM ++ paper</a>. They’re computing an average (aka. smoothed) sensitivity map over a small neighborhood of the input image. They make this “neighborhood” by creating multiple noisy versions of the input.</p>
<aside class="quote no-group" data-username="Linus" data-post="5" data-topic="116601">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/f4b2a3/48.png" class="avatar"> Linus:</div>
<blockquote>
<p>n my case, it seems that the noise is dominating and more or less the only thing I see is something related to this noise.</p>
</blockquote>
</aside>
<p>If the noise is dominating, then there might something wrong with your transformations. I would ensure whatever transformations you’re using are appropriate for the CNN model you’re using (looks like ResNet). Also, make sure you scale any images to <code>[0, 1]</code> (or [0, 255]`) for visualizing using matplotlib.</p>
<p>But honestly, there are so many things that could cause your problem, and it’s just a guessing game (at least it is for me) unless we have a reproducible example. Maybe you could make and share a minimal example on Google Colab? Looks like you’re already using Colab, so hopefully not too much of a problem to create a simple example that doesn’t require access to files in your Google Drive, etc.</p>
<aside class="quote no-group" data-username="Linus" data-post="5" data-topic="116601">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/f4b2a3/48.png" class="avatar"> Linus:</div>
<blockquote>
<p>Can I turn the noise completely off?</p>
</blockquote>
</aside>
<p>If you want a method without random additive noise, use <code>GradCAMpp</code> instead of <code>SmoothGradCAMpp</code>.</p>

Linus: <p>You are much more accurate in your guess than you think - the solution was to use GradCAM! This makes the feature maps repetable and I can now investigate them!</p>
<p>Big thanks to you for your fast and very helpful  insights!</p>