Visualize feature map

oasjd7: <p>Hi, all.<br>
I have some questions about the visualization.<br>
I`m newbie in this field…so maybe this is silly questions.</p>
<p>I have MNIST dataset. and I want to visualize the output of my encoder.<br>
<code>(Input: MNIST data) -&gt; MY_ENCODER -&gt; output -&gt; visualization.</code></p>
<ol>
<li>
<p>How can I visualize the data from output of CNN ?</p>
</li>
<li>
<p>If I use MNIST dataset as input to my encoder, can I use the output of this encoder to re-construct<br>
image like original MNIST ?<br>
If this is impossible, do I have to use re-construction loss with decoder ?</p>
</li>
<li>
<p>Is there any support codes to visualize feature map ?</p>
</li>
</ol>

ptrblck: <ol>
<li>
<p>You can just use a plot library like <code>matplotlib</code> to visualize the output.</p>
</li>
<li>
<p>Sure! You could use some loss function like <code>nn.BCELoss</code> as your criterion to reconstruct the images.</p>
</li>
<li>
<p>Forward hooks are a good choice to get the activation map for a certain input.</p>
</li>
</ol>
<p>Here is a small code example as a starter:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader

import torchvision.transforms as transforms
import torchvision.datasets as datasets

import matplotlib.pyplot as plt


class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(1, 3, 3, 1, 1)
        self.pool1 = nn.MaxPool2d(2)
        self.conv2 = nn.Conv2d(3, 6, 3, 1, 1)
        self.pool2 = nn.MaxPool2d(2)
        
        self.conv_trans1 = nn.ConvTranspose2d(6, 3, 4, 2, 1)
        self.conv_trans2 = nn.ConvTranspose2d(3, 1, 4, 2, 1)
        
    def forward(self, x):
        x = F.relu(self.pool1(self.conv1(x)))
        x = F.relu(self.pool2(self.conv2(x)))        
        x = F.relu(self.conv_trans1(x))
        x = self.conv_trans2(x)
        return x

dataset = datasets.MNIST(
    root='PATH',
    transform=transforms.ToTensor()
)
loader = DataLoader(
    dataset,
    num_workers=2,
    batch_size=8,
    shuffle=True
)

model = MyModel()
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

epochs = 1
for epoch in range(epochs):
    for batch_idx, (data, target) in enumerate(loader):
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, data)
        loss.backward()
        optimizer.step()
        
        print('Epoch {}, Batch idx {}, loss {}'.format(
            epoch, batch_idx, loss.item()))


def normalize_output(img):
    img = img - img.min()
    img = img / img.max()
    return img

# Plot some images
idx = torch.randint(0, output.size(0), ())
pred = normalize_output(output[idx, 0])
img = data[idx, 0]

fig, axarr = plt.subplots(1, 2)
axarr[0].imshow(img.detach().numpy())
axarr[1].imshow(pred.detach().numpy())

# Visualize feature maps
activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model.conv1.register_forward_hook(get_activation('conv1'))
data, _ = dataset[0]
data.unsqueeze_(0)
output = model(data)

act = activation['conv1'].squeeze()
fig, axarr = plt.subplots(act.size(0))
for idx in range(act.size(0)):
    axarr[idx].imshow(act[idx])
</code></pre>

oasjd7: <p>Thanks for help newbie…!</p>
<p>I have some questions again…(sorry to bother you)</p>
<ol>
<li>
<p>In your code, <code>MyModel()</code> looks like just kinds of model.<br>
but I can’t distinguish between classifier model or reconstruction model.<br>
How can this model do works for re-construct ?</p>
</li>
<li>
<p>I have encoder model and I want to visualize what the results of this encoder generate reconstruction images, then what should I add components to reconstruct images?</p>
</li>
<li>
<p><code>get_activation()</code> function in your code, this output is looks like digit. I think this is output of first conv but how can first conv generate like digit.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3565f05717202c7ec505d674f17cf0077a049df6.png" alt="fig2" data-base62-sha1="7CnHWSXsYZaHFnAMWR7mRD6hvKu" width="297" height="434"><br>
I expected image like those feature maps.<br>
I’m confused about feature map…T.T<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/09a4f74777539ac7fbf0715e6cdb9415a30eb730.jpeg" alt="feature" data-base62-sha1="1njJzNe7prM9wMwOCVOcAdhsMRa" width="347" height="145"></p>
</li>
</ol>

ptrblck: <ol>
<li>
<p>The model I created is reconstructing the images just by its architecture. As you can see I’ve created a “bottleneck” in the model, i.e. the activations will get smaller, and after it I used transposed conv layers to increase the spatial size again.<br>
The last layer outputs the same shape as the input had. While this simple model is working on the MNIST dataset, it might be too simple for other more complicated models.</p>
</li>
<li>
<p>You could need to add the decoding part, i.e. creating output images out of your latent vector. Depending on your encoder’s output, you might need to reshape it. If you need some help implementing it, could you post your encoder architecture?</p>
</li>
<li>
<p>Yes, you are currently visualizing the activations, i.e. the output of intermediate layers. In case you want to visualize the kernels directly, you could use the following code:</p>
</li>
</ol>
<pre><code class="lang-python"># Visualize conv filter
kernels = model.conv1.weight.detach()
fig, axarr = plt.subplots(kernels.size(0))
for idx in range(kernels.size(0)):
    axarr[idx].imshow(kernels[idx].squeeze())
</code></pre>

oasjd7: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Here is my encoder model. I need your help to get reconstruct image.<br>
and I have generated filter image with your code then how can I generate feature map from this filter figure?</p>
<pre><code class="lang-auto">class Extractor(nn.Module):
    def __init__(self):
        super(Extractor, self).__init__()
        self.extractor = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2),

            nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5, stride=1, padding=0),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2),

            nn.Conv2d(in_channels=128, out_channels=256, kernel_size=5, stride=1, padding=0),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2),
        )

    def forward(self, x):
        x = self.extractor(x)
        x = x.view(-1, out_size)
        return x
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/33cd1667337d11fb60f120eb1300c8b562aa5676.png" alt="fig_filter" data-base62-sha1="7ofKr0sIbyUzCgM8qWu6PNHhj9Q" width="640" height="480"></p>

ptrblck: <p>Based on your architecture, a decoder could look like this:</p>
<pre><code class="lang-python">class Extractor(nn.Module):
    def __init__(self):
        super(Extractor, self).__init__()
        self.extractor = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2),

            nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5, stride=1, padding=0),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2),

            nn.Conv2d(in_channels=128, out_channels=256, kernel_size=5, stride=1, padding=0),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2),
        )
        
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(256, 128, 5, 2),
            nn.ReLU(),
            nn.ConvTranspose2d(128, 64, 6, 2),
            nn.ReLU(),
            nn.ConvTranspose2d(64, 3, 6, 2)
        )

    def forward(self, x):
        x = self.extractor(x)
        x = self.decoder(x)
        return x
</code></pre>
<p>To visualize the activations, you could try the same code as above:</p>
<pre><code class="lang-python">model = Extractor()
model.extractor[0].register_forward_hook(get_activation('ext_conv1'))
x = torch.randn(1, 3, 96, 96)
output = model(x)
print(output.shape)
&gt; torch.Size([1, 3, 96, 96])

act = activation['ext_conv1'].squeeze()
num_plot = 4
fig, axarr = plt.subplots(min(act.size(0), num_plot))
for idx in range(min(act.size(0), num_plot)):
    axarr[idx].imshow(act[idx])
</code></pre>
<p>I’m just plotting the first 4 maps, so you could just remove <code>num_plot</code> and the <code>min</code> call if you want to plot all maps.</p>

oasjd7: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
cool !</p>
<p>Now I can get filter image.(3x3 in you examples)<br>
How can I apply this filter to input image?<br>
To get feature map like this .<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/09a4f74777539ac7fbf0715e6cdb9415a30eb730.jpeg" alt="feature" data-base62-sha1="1njJzNe7prM9wMwOCVOcAdhsMRa" width="347" height="145"></p>

ptrblck: <p>The image you’ve posted is from a <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf">Krizhevsky et al.</a> and shows the learned filter kernels.<br>
You are not seeing a feature map, but 96 kernels of size <code>3x11x11</code>.<br>
To get a similar image, you can use this code snippet:</p>
<pre><code class="lang-python">from torchvision.utils import make_grid

kernels = model.extractor[0].weight.detach().clone()
kernels = kernels - kernels.min()
kernels = kernels / kernels.max()
img = make_grid(kernels)
plt.imshow(img.permute(1, 2, 0))
</code></pre>

1414b35e42c77e0a57dd: <p>How can get feature map not kernels ?</p>

ptrblck: <p>Would this approach work?</p>
<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="29597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>To visualize the activations, you could try the same code as above:</p>
<pre><code class="lang-auto">model = Extractor()
model.extractor[0].register_forward_hook(get_activation('ext_conv1'))
x = torch.randn(1, 3, 96, 96)
output = model(x)
print(output.shape)
&gt; torch.Size([1, 3, 96, 96])

act = activation['ext_conv1'].squeeze()
num_plot = 4
fig, axarr = plt.subplots(min(act.size(0), num_plot))
for idx in range(min(act.size(0), num_plot)):
    axarr[idx].imshow(act[idx])
</code></pre>
<p>I’m just plotting the first 4 maps, so you could just remove <code>num_plot</code> and the <code>min</code> call if you want to plot all maps.</p>
</blockquote>
</aside>

wiiiktor: <p>In the line:<br>
img = data[idx, 0]<br>
array “data” is referenced, although it is not defined before! - code is working ok, but how?..</p>

ptrblck: <p><code>data</code> and <code>output</code> are both defined in the training for loop.<br>
I’m just reusing them for visualization.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> how we can display output of layer in the original size of image. for example in UNet  layer up2 (decoder section) the torch feature output size is <code>torch.Size([1, 128, 120, 160])</code> how can I display it on the original size of image which is [1, 240, 320]?</p>
<p>Actually, I posted same question in separate thread <a href="https://discuss.pytorch.org/t/how-visualise-feature-map-in-original-size-of-input/39778">https://discuss.pytorch.org/t/how-visualise-feature-map-in-original-size-of-input/39778</a></p>

adamdayan: <p>This is some super helpful code for me! One question I have is whether the activation captured by the hook is pre or post the application of the ReLU function? Thanks!</p>

ptrblck: <p>It would be pre-ReLU based on the registered hook.<br>
However, since <code>self.extractor</code> uses inplace <code>nn.ReLU</code>s after the conv layers, the relu will be applied on the stored activation.</p>

mvadrev: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>:</p>
<p>What are there images in the link? <a href="https://towardsdatascience.com/how-to-visualize-convolutional-features-in-40-lines-of-code-70b7d87b0030" rel="nofollow noopener">https://towardsdatascience.com/how-to-visualize-convolutional-features-in-40-lines-of-code-70b7d87b0030</a></p>
<p>Are they activation maps or kernels?</p>

Denis_Dmitriev: <p>Feature map visualization: <a href="https://youtu.be/RNnKtNrsrmg" rel="noopener nofollow ugc">https://youtu.be/RNnKtNrsrmg</a><br>
Some images from the video:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1ec345222b429594be43362b565a6ee5a4cd4a6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c1ec345222b429594be43362b565a6ee5a4cd4a6" title="4blk"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1ec345222b429594be43362b565a6ee5a4cd4a6_2_690x388.jpeg" alt="4blk" data-base62-sha1="rFwdtsYbGYdOQPczA2RYRDqFKtM" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1ec345222b429594be43362b565a6ee5a4cd4a6_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1ec345222b429594be43362b565a6ee5a4cd4a6_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1ec345222b429594be43362b565a6ee5a4cd4a6_2_1380x776.jpeg 2x" data-dominant-color="26222F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4blk</span><span class="informations">2335×1315 537 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

raj_patel: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I’m learning PyTorch. I tried your code for the activation but I got an error.</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-71-94fc5c43ff92&gt; in &lt;module&gt;()
     12 axarr[0].imshow(img.detach().numpy())
     13 # print(pred.detach().numpy().shape)
---&gt; 14 axarr[1].imshow(pred.detach().numpy())
     15 # Visualize feature maps
     16 activation = {}

4 frames
/usr/local/lib/python3.6/dist-packages/matplotlib/image.py in set_data(self, A)
    688                 or self._A.ndim == 3 and self._A.shape[-1] in [3, 4]):
    689             raise TypeError("Invalid shape {} for image data"
--&gt; 690                             .format(self._A.shape))
    691 
    692         if self._A.ndim == 3:

TypeError: Invalid shape () for image data
</code></pre>
<p>I’m working on cifar 10 dataset and shape of my training data<br>
X_train_torch is (50000, 3, 32, 32)</p>
<p>Thanks!</p>

ptrblck: <p>Could you post the shape of <code>pred</code>?<br>
If it’s more than a single prediction, you should index it first, since <code>imshow</code> can only visualize a single array.</p>

raj_patel: <p>Why the shape of pred is empty?</p>
<pre><code class="lang-auto">print(pred.shape)
&gt;&gt;&gt;torch.Size([])
</code></pre>