How to predict the matrix before the last layer?

username: namduc
cooked: <p>I have trained model with layers stacks in nn.Sequential for classification problem.<br>
The ConvNet architecture look like this:</p>
<pre><code class="lang-auto">class ConvNet(nn.Module):
    def __init__(self,num_classes=8):
        super(ConvNet,self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1,64,kernel_size=7),                                           
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.AvgPool2d(kernel_size=2,stride=2)                                 
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(64,128,kernel_size=7,stride=2),                               
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.AvgPool2d(kernel_size=2,stride=2)                                     
        )
        self.layer3 = nn.Sequential(
            nn.Conv2d(128,256,kernel_size=3),                                        
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.AvgPool2d(kernel_size=2,stride=2)                                 
        )
        self.layer4 = nn.Sequential(                                                
            nn.Conv2d(256,512,kernel_size=3),                                     
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.AvgPool2d(kernel_size=2,stride=2),                                   
            nn.BatchNorm2d(512)                                     
        )
        self.hidden = nn.Linear(2*2*512,1024)
        self.drop = nn.Dropout(0.6)
        self.dense1 = nn.Sequential(
            nn.Linear(1024,256),
            nn.ReLU(),
            nn.Dropout(0.25)
        )
        self.dense2 = nn.Sequential(
            nn.Linear(256,64),
            nn.ReLU()
        )
        self.fc1 = nn.Linear(64,32)
        self.fc = nn.Linear(32,num_classes)

    def forward(self,x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = out.reshape(out.size(0),-1)
        out = F.relu(self.hidden(out))
        out = self.drop(out)
        out = self.dense1(out)
        out = self.dense2(out)
        out = F.relu(self.fc1(out))
        out = self.fc(out)
        return out
</code></pre>
<p><em>ConvNet(<br>
(layer1): Sequential(<br>
(0): Conv2d(1, 64, kernel_size=(7, 7), stride=(1, 1))<br>
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): AvgPool2d(kernel_size=2, stride=2, padding=0)<br>
)<br>
(layer2): Sequential(<br>
(0): Conv2d(64, 128, kernel_size=(7, 7), stride=(2, 2))<br>
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): AvgPool2d(kernel_size=2, stride=2, padding=0)<br>
)<br>
(layer3): Sequential(<br>
(0): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1))<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): AvgPool2d(kernel_size=2, stride=2, padding=0)<br>
)<br>
(layer4): Sequential(<br>
(0): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1))<br>
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): AvgPool2d(kernel_size=2, stride=2, padding=0)<br>
(4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(hidden): Linear(in_features=2048, out_features=1024, bias=True)<br>
(drop): Dropout(p=0.6, inplace=False)<br>
(dense1): Sequential(<br>
(0): Linear(in_features=1024, out_features=256, bias=True)<br>
(1): ReLU()<br>
(2): Dropout(p=0.25, inplace=False)<br>
)<br>
(dense2): Sequential(<br>
(0): Linear(in_features=256, out_features=64, bias=True)<br>
(1): ReLU()<br>
)<br>
(fc1): Linear(in_features=64, out_features=32, bias=True)<br>
(fc): Linear(in_features=32, out_features=8, bias=True)<br>
)</em></p>
<h4>Then, I have used this method delete the last layer in order to obtain the matrix 32:</h4>
<pre><code class="lang-auto">model = ConvNet(8).to(device)
model.load_state_dict(torch.load('model.pt'))
removed = list(model.children())[:-1]
new_model= torch.nn.Sequential(*removed)
print(new_model)
</code></pre>
<p><em>Output:<br>
Sequential(<br>
(0): Sequential(<br>
(0): Conv2d(1, 64, kernel_size=(7, 7), stride=(1, 1))<br>
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): AvgPool2d(kernel_size=2, stride=2, padding=0)<br>
)<br>
(1): Sequential(<br>
(0): Conv2d(64, 128, kernel_size=(7, 7), stride=(2, 2))<br>
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): AvgPool2d(kernel_size=2, stride=2, padding=0)<br>
)<br>
(2): Sequential(<br>
(0): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1))<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): AvgPool2d(kernel_size=2, stride=2, padding=0)<br>
)<br>
(3): Sequential(<br>
(0): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1))<br>
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): AvgPool2d(kernel_size=2, stride=2, padding=0)<br>
(4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(4): Linear(in_features=2048, out_features=1024, bias=True)<br>
(5): Dropout(p=0.6, inplace=False)<br>
(6): Sequential(<br>
(0): Linear(in_features=1024, out_features=256, bias=True)<br>
(1): ReLU()<br>
(2): Dropout(p=0.25, inplace=False)<br>
)<br>
(7): Sequential(<br>
(0): Linear(in_features=256, out_features=64, bias=True)<br>
(1): ReLU()<br>
)<br>
(8): Linear(in_features=64, out_features=32, bias=True)<br>
)</em><br>
However, when I want to predict new picture to get matrix 32, I get an error:</p>
<h4>RuntimeError: size mismatch, m1: [1024 x 2], m2: [2048 x 1024] at C:/w/1/s/tmp_conda_3.8_075429/conda/conda-bld/pytorch_1579852542185/work/aten/src\THC/generic/THCTensorMathBlas.cu:290</h4>
<p>Something was wrong here?<br>
How can i get the maxtrix 32 before last layer classifies?</p>

username: ptrblck
cooked: <p>I assume you’ve tried to create the new model by wrapping the child modules into an <code>nn.Sequential</code> container.<br>
If that’s the case, note that you will lose all functional API calls from the <code>forward</code> method in your original model, e.g. <code>out = out.reshape(out.size(0),-1)</code> as well as the <code>F.relu</code> calls.<br>
Thus you should add them via e.g. <code>nn.Flatten</code>, <code>nn.ReLU</code>.</p>
<p>Alternatively, you could manipulate the <code>forward</code> method or use forward hooks to get the desired activation.</p>

username: namduc
cooked: <p>Can I use this method for replace <code>out = out.reshape(out.size(0),-1)</code> ?</p>
<pre><code class="lang-auto">class Flatten(nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)
</code></pre>
<p>Then, forward function look like this:</p>
<pre><code class="lang-auto">def forward(self,x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.layer3(out)
    out = self.layer4(out)
    out = Flatten(out)
    out = nn.ReLU(self.hidden(out))
    out = self.drop(out)
    out = self.dense1(out)
    out = self.dense2(out)
    out = nn.ReLU(self.fc1(out))
    out = self.fc(out)
    return out
</code></pre>
<p>Is this right ?</p>

username: ptrblck
cooked: <p>Generally yes, but you would have to create instances of these layers or use the functional API in your <code>forward</code> method:</p>
<pre><code class="lang-python">def forward(self,x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.layer3(out)
    out = self.layer4(out)
    out = Flatten()(out)
    out = nn.ReLU()(self.hidden(out))
    out = self.drop(out)
    out = self.dense1(out)
    out = self.dense2(out)
    out = nn.ReLU()(self.fc1(out))
    out = self.fc(out)
    return out
</code></pre>

username: namduc
cooked: <p>I have followed your <code>forward</code> function, but I still get the same error.<br>
I called all parameter in file test.py  like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
class Flatten(nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)

class ConvNet(nn.Module):
    def __init__(self,num_classes=8):
        super(ConvNet,self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1,64,kernel_size=7),                                          
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.AvgPool2d(kernel_size=2,stride=2)                                     
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(64,128,kernel_size=7,stride=2),                               
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.AvgPool2d(kernel_size=2,stride=2)                                    
        )
        self.layer3 = nn.Sequential(
            nn.Conv2d(128,256,kernel_size=3),                                        
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.AvgPool2d(kernel_size=2,stride=2)                                    
        )
        self.layer4 = nn.Sequential(                                                
            nn.Conv2d(256,512,kernel_size=3),                                        
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.AvgPool2d(kernel_size=2,stride=2),                                   
            nn.BatchNorm2d(512)                                     
        )
        self.hidden = nn.Linear(2*2*512,1024)
        self.drop = nn.Dropout(0.6)
        self.dense1 = nn.Sequential(
            nn.Linear(1024,256),
            nn.ReLU(),
            nn.Dropout(0.25)
        )
        self.dense2 = nn.Sequential(
            nn.Linear(256,64),
            nn.ReLU()
        )
        self.fc1 = nn.Linear(64,32)
        self.fc = nn.Linear(32,num_classes)

    def forward(self,x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = Flatten()(out)
        out = nn.ReLU()(self.hidden(out))
        out = self.drop(out)
        out = self.dense1(out)
        out = self.dense2(out)
        out = nn.ReLU()(self.fc1(out))
        out = self.fc(out)
        return out

model = ConvNet(8).to(device)
model.load_state_dict(torch.load('model.pt'))
removed = list(model.children())[:-1]
new_model= torch.nn.Sequential(*removed)
with torch.no_grad():
     image_trans = torch.from_numpy(test_image.astype(np.float32)).to(device)
     prediction = new_model(image_trans)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test.py", line 121, in &lt;module&gt;
    prediction = new_model(image_trans)
  File "C:\ProgramData\Anaconda3\envs\deep\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\deep\lib\site-packages\torch\nn\modules\container.py", line 100, in forward
    input = module(input)
  File "C:\ProgramData\Anaconda3\envs\deep\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)*
  File "C:\ProgramData\Anaconda3\envs\deep\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\ProgramData\Anaconda3\envs\deep\lib\site-packages\torch\nn\functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: size mismatch, m1: [1024 x 2], m2: [2048 x 1024] at C:/w/1/s/tmp_conda_3.8_075429/conda/conda-bld/pytorch_1579852542185/work/aten/src\THC/generic/THCTensorMathBlas.cu:290
</code></pre>

username: ptrblck
cooked: <p>Sorry for being not clear enough.</p>
<p>You won’t be able to copy the child modules directly to an <code>nn.Sequential</code> container, as all functional calls as well as locally defined modules in the original <code>forward</code> function will be missing.</p>
<p>Also, <code>removed</code> contains the order of modules as they were created in the <code>__init__</code>.<br>
If this order is wrong, also the order of modules in the sequential container will be wrong.</p>
<p>To add the <code>Flatten</code> and <code>ReLU</code> modules into a sequential container, you could use an approach similar to this one:</p>
<pre><code class="lang-python">new_model= torch.nn.Sequential(
    *(list(removed[:4]) + [nn.Flatten(), nn.ReLU()] + list(removed[5:7]) + [nn.ReLU()] + list(removed[-2:-1])))
</code></pre>
<p>Note, that the order is currently wrong and the model won’t work.</p>
<p><code>nn.Sequential</code> is used for simple models and as you can see, manipulating the forward method is easier by writing a custom module and defining the <code>forward</code> manually.</p>

username: namduc
cooked: <p>Thanks, i understand.<br>
One more question, if forward function look like this:</p>
<pre><code class="lang-auto">   def forward(self,x):
        out = self.layer1(x)
        out = nn.Flatten()(out)
</code></pre>
<p>And sequential container look like this:</p>
<pre><code class="lang-auto">nn.Sequential(layer[0] + [nn.Flatten()])
</code></pre>
<p>or like this:</p>
<pre><code class="lang-auto">nn.Sequential( [nn.Flatten()] + layer[0])
</code></pre>
<p>Which one is right ?<br>
I mean, I have problems with the order of using the functional APIs in  <code>nn.Sequential</code></p>

username: ptrblck
cooked: <p>For a single layer, this approach would work.<br>
However, note that <code>model.children()</code> returns the child modules in the order they were initialized in the <code>__init__</code>, which might not be the same order they are called in the <code>forward</code>.</p>

username: namduc
cooked: <p>I have a different question<br>
I trained the model with batch_size = 8<br>
However I want to test with one single image on the model obtained.<br>
How do i achieve this without having to retrain the model with batch_size = 1?</p>

username: ptrblck
cooked: <p>You don’t have to retrain the model, but just set it to evaluation mode via <code>model.eval()</code>.<br>
This will make sure to e.g. disable dropout and use the estimated stats in batchnorm layers instead of the batch stats.</p>