Display feature maps of MNAS

username: Heyam_Bin_Jahlan
cooked: <p>Hi every one;</p>
<p>I need help, please. I have two images then I take the difference between these two images into one single image. I use Pretrained Mobile Neural Architecture Search (MNAS) to extract features only. I need to display the feature map of this image from the middle Conv layer of MNAS and the feature map of the final Conv layer before classifier.</p>
<p>I tried this snipped from <a class="mention" href="/u/ptrblck">@ptrblck</a> but I have a problem.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torchvision import models
import torchvision.transforms as transforms
import torchvision.datasets as datasets

import matplotlib.pyplot as plt
from PIL import Image
import numpy as np


class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = models.mnasnet1_0(pretrained=True)
        self.convNet = nn.Sequential (self.conv1)
        self.convNet2= self.conv1.classifier=nn.Identity()
        
    def forward(self, x):
        
        x = self.convNet(x)
        return x

img1 = './dataset/11/frame001.jpg'
img2=  './dataset/11/frame004.jpg'
img1 = Image.open(str(img1))
img2 = Image.open(str(img2)) 
img1= transform=transforms.ToTensor()(img1)
img2= transform=transforms.ToTensor()(img2)


model = MyModel()


# Visualize feature maps
activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model.conv1.register_forward_hook(get_activation('conv1'))

img= torch.abs(img1 - img2)
img.unsqueeze_(0)
output = model(img)


act = activation['conv1'].squeeze()
fig, axarr = plt.subplots(act.size(0))
plt.imshow(act)
</code></pre>
<p>I get this error</p>
<pre><code class="lang-auto">
  File "C:\Users\Windows10\anaconda3\envs\Heyam\lib\site-packages\matplotlib\axes\_axes.py", line 5523, in imshow
    im.set_data(X)

  File "C:\Users\Windows10\anaconda3\envs\Heyam\lib\site-packages\matplotlib\image.py", line 709, in set_data
    raise TypeError("Invalid shape {} for image data"

TypeError: Invalid shape (1280,) for image data
</code></pre>

username: ptrblck
cooked: <p>Your activation (<code>act</code>) seems to have a wrong size and <code>plt.imshow</code> cannot visualize it.<br>
Make sure you are trying to plot a tensor in the shape <code>[height, width, 3]</code> or <code>[height, width]</code> and pass it as a numpy array to <code>plt.imshow</code> via <code>tensor.numpy()</code>.</p>

username: Heyam_Bin_Jahlan
cooked: <p>thanks alot.<br>
act shape</p>
<pre><code class="lang-auto">torch.Size([1280])
</code></pre>
<p>can you help how to change the shape of the act tensor ?</p>

username: ptrblck
cooked: <p>The activation seems to be the output activation of the <code>mnasnet1_0</code>, which would be the output logits.<br>
If you want to visualize them, you could use e.g. <code>plt.plot</code> instead of <code>plt.imshow</code>, since the activation is not an image but just a flattened tensor containing the class logits.<br>
Alternatively, you could reshape the activation to the aforementioned shape.</p>

username: Heyam_Bin_Jahlan
cooked: <p>I tired of this error. can you help me please. <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
My code</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader

import torchvision.transforms as transforms
import torchvision.datasets as datasets
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
import numpy as np
from torchvision import models
import torchvision.transforms as transforms
import cv2


class MyModel(nn.Module):
   def __init__(self):
       super(MyModel, self).__init__()
       self.mod= models.mnasnet1_0(pretrained=True)
       self.convNet= nn.Sequential(*list(self.mod.children()))

       
   def forward(self, x):

       seqLen = x.size(0) - 1
       for t in range(0, seqLen):
           x1 = x[t] - x[t+1]
           x2 = self.convNet(x1)

img1 = './dataset/11/frame001.jpg'
img2=  './dataset/11/frame004.jpg'
img1 = Image.open(str(img1))
img2 = Image.open(str(img2)) 




trans= transforms.Compose([transforms.RandomResizedCrop(224),
                                               transforms.ToTensor(),
                                               transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])


inpSeq = []
# img1=Image.fromarray(img1)
# img2=Image.fromarray(img2)

inpSeq.append(trans(img1.convert('RGB')))
inpSeq.append(trans(img2.convert('RGB')))
inpSeq = torch.stack(inpSeq, 0)

inpSeq=inpSeq.unsqueeze(0)
model = MyModel()


activation = {}
def get_activation(name):
   def hook(model, input, output):
       activation[name] = output.detach()
   return hook

model.convNet.register_forward_hook(get_activation('conv1'))

output = model(inpSeq)

act = activation['conv1'].squeeze()
plt.imshow(act[0])
</code></pre>
<p>error is here:</p>
<pre><code class="lang-auto">Traceback (most recent call last):

  File "C:\Users\Windows10\Desktop\Code1\exp.py", line 67, in &lt;module&gt;
    act = activation['conv1'].squeeze()

KeyError: 'conv1'
</code></pre>

username: ptrblck
cooked: <p>If I’m using a <code>seqLen</code> of <code>1</code> and a batch size of <code>2</code>, the internal loop will not be executed and thus no hooks are called:</p>
<pre><code class="lang-python">x = torch.randn(1, 2, 3, 224, 224)
model(x)
</code></pre>
<p>since your loop would be:</p>
<pre><code class="lang-python">for t in range(0, 0):
</code></pre>
<p>and would thus not execute the <code>self.convNet</code>.<br>
Note that your current <code>forward</code> method also doesn’t return anything, so you might want to change that as well.</p>
<p>However, if I’m using a <code>seqLen</code> of <code>2</code>, I get a shape mismatch error:</p>
<pre><code class="lang-python">x = torch.randn(2, 2, 3, 224, 224)
output = model(x)
&gt; RuntimeError: mat1 and mat2 shapes cannot be multiplied (17920x7 and 1280x1000)
</code></pre>
<p>so I guess you might be facing the first issue.</p>
<p>The shape mismatch error is most likely raised, since you are rewrapping the child modules in an <code>nn.Sequential</code> container and are thus dropping the functional calls from the <a href="https://github.com/pytorch/vision/blob/80f41f8d32b1fcb380d5df2116063af7034ff29a/torchvision/models/mnasnet.py#L147-L151">original forward</a>.<br>
I would not recommend to wrap arbitrary models into <code>nn.Sequential</code> unless you are sure that all modules are just executed in a sequential way without any conditions, functional calls, loops etc.</p>
<p>If I avoid the shape mismatch error and use:</p>
<pre><code class="lang-python">def forward(self, x):
   seqLen = x.size(0) - 1
   for t in range(0, seqLen):
       x1 = x[t] - x[t+1]
       x2 = self.mod(x1)
</code></pre>
<p>with <code>x = torch.randn(2, 2, 3, 224, 224)</code>, the forward hook is properly called and <code>activation</code> will be populated.</p>