Why removing last layer is causing size mismatch?

username: mmisiur
cooked: <p>Hi there,<br>
I’m quite new to torch, so maybe it’s sth simple. I’ve trained my net, which is:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool = nn.MaxPool2d(4, 4)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(44896, 4096)
        self.fc2 = nn.Linear(4096, 1024)
        self.fc3 = nn.Linear(1024, 251)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(x.size(0), 44896)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return 
</code></pre>
<p>And now I want to remove last layer (fc3), to make a feature extractor out of my net. So I’m loading the trained model, and removing last layer:</p>
<pre><code class="lang-auto">    #loading model
    model = torch.load('./net_100_test_train')
    model.eval()

    #loading, preparing and normalizing a test image
    img=mpimg.imread('./0298_0040_0001.png')
    img = np.array(img*2-1)
    img = np.tile(img, (1, 1, 1))
    img = torch.from_numpy(img)
    img = torch.unsqueeze(img, 1)

    #removing last layer
    new_model = nn.Sequential(*list(model.children())[:-1])
    with torch.no_grad():
        output = new_model(img.to(device))
</code></pre>
<p>which is giving me an error:</p>
<pre><code class="lang-auto">RuntimeError: size mismatch, m1: [1952 x 93], m2: [44896 x 4096] at c:\a\w\1\s\windows\pytorch\aten\src\thc\generic/THCTensorMathBlas.cu:266
</code></pre>
<p>Is this because I’m using Sequential? How should I remove last layer properly, and be able to extract features?</p>
<p>edit:<br>
I think I workaround it by replacing the last layer with an Identity layer, but the layer itself is still there.</p>

username: ptrblck
cooked: <p>If you try to create a new <code>nn.Sequential</code> model using the layers from <code>Net</code>, the flatten operation will be missing.<br>
Additionally, you are currently using only one pooling layer, as both are defined as <code>self.pool</code>.</p>
<p>Probably the most straightforward way would be to derive another class from <code>Net</code> and just manipulate the <code>forward</code> method so that it returns your desired features tensor.</p>
<p>However, if you would like to use the <code>nn.Sequential</code> approach, you could define a <code>Flatten</code> layer and add it to the right place:</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool1 = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(44896, 4096)
        self.fc2 = nn.Linear(4096, 1024)
        self.fc3 = nn.Linear(1024, 251)

    def forward(self, x):
        x = self.pool1(F.relu(self.conv1(x)))
        x = self.pool2(F.relu(self.conv2(x)))
        print(x.shape)
        x = x.view(x.size(0), 44896)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return 


x = torch.randn(1, 1, 259, 199)
model = Net()
output = model(x)

class Flatten(nn.Module):
    def __init__(self):
        super(Flatten, self).__init__()
        
    def forward(self, x):
        return x.view(x.size(0), -1)

new_model = nn.Sequential(*[*list(model.children())[:4], Flatten(), *list(model.children())[4:-1]])
new_model(x)
</code></pre>

username: mmisiur
cooked: <p>Thanks!<br>
I’ll probably go with the <code>Flatten</code> solution.<br>
Indeed I defined theese pools wrong, thanks for finding that! I’ve retrained the net with correct pooling and, interestingly, the accuracy has improved.</p>

username: amlandas78
cooked: <p>I am trying to remove the 4th maxpooling layer in VGG16 using</p>
<pre><code class="lang-auto">    list1 = list(vgg.classifier._modules.values())[:23]
    list1.extend(list(vgg.classifier._modules.values())[24:-1])

    list2 = list(vgg.features._modules.values())[:23]
    list2.extend(list(vgg.features._modules.values())[24:-1])
    vgg.classifier = nn.Sequential(*list1)
    self.RCNN_base = nn.Sequential(*list2)
</code></pre>
<p>I have also changed</p>
<pre><code class="lang-auto">self.RCNN_roi_pool = _RoIPooling(cfg.POOLING_SIZE, cfg.POOLING_SIZE, 1.0/8.0)
</code></pre>
<p>from</p>
<pre><code class="lang-auto">self.RCNN_roi_pool = _RoIPooling(cfg.POOLING_SIZE, cfg.POOLING_SIZE, 1.0/16.0)
</code></pre>
<p>However I am getting size mismatch error</p>
<pre><code class="lang-auto">RuntimeError: size mismatch, m1: [256 x 1000], m2: [4096 x 8] at /opt/conda/conda-bld/pytorch_1524577177097/work/aten/src/THC/generic/THCTensorMathBlas.cu:249
</code></pre>
<p>Can you please help me with this?</p>

username: ptrblck
cooked: <p>Could you post a code snippet to reproduce this issue?<br>
I’m not sure, where and hwo <code>self.RCNN_poi_pool</code> is defined in your model etc.</p>

username: amlandas78
cooked: <p>I am using <a href="https://github.com/DebasmitaGhose/Multimodal_Influenza_Detection/tree/master/faster-rcnn.pytorch" rel="nofollow noopener">https://github.com/DebasmitaGhose/Multimodal_Influenza_Detection/tree/master/faster-rcnn.pytorch</a>.</p>
<p>In order to remove the 4th max pooling layer I modified the lib/model/vgg16.py as follows</p>
<pre><code class="lang-auto">    list1 = list(vgg.classifier._modules.values())[:23]
    list1.extend(list(vgg.classifier._modules.values())[24:-1])

    list2 = list(vgg.features._modules.values())[:23]
    list2.extend(list(vgg.features._modules.values())[24:-1])
    vgg.classifier = nn.Sequential(*list1)
    self.RCNN_base = nn.Sequential(*list2)
</code></pre>
<p>Moreover in <a href="https://github.com/DebasmitaGhose/Multimodal_Influenza_Detection/blob/master/faster-rcnn.pytorch/lib/model/faster_rcnn/faster_rcnn.py" rel="nofollow noopener">https://github.com/DebasmitaGhose/Multimodal_Influenza_Detection/blob/master/faster-rcnn.pytorch/lib/model/faster_rcnn/faster_rcnn.py</a>, I modified the following lines</p>
<pre><code class="lang-auto">self.RCNN_roi_pool = _RoIPooling(cfg.POOLING_SIZE, cfg.POOLING_SIZE, 1.0/8.0)
self.RCNN_roi_align = RoIAlignAvg(cfg.POOLING_SIZE, cfg.POOLING_SIZE, 1.0/8.0)
</code></pre>
<p>In the lib/model/utils/config.py I made the following modifications</p>
<pre><code class="lang-auto">__C.DEDUP_BOXES = 1. / 16.
__C.FEAT_STRIDE = [16, ]
</code></pre>
<p>to</p>
<pre><code class="lang-auto">__C.DEDUP_BOXES = 1. / 8.
__C.FEAT_STRIDE = [8, ]
</code></pre>