How to resolve - RuntimeError: size mismatch, m1: [100 x 228], m2: [152 x 36]

username: Waqas
cooked: <p>Hi!</p>
<p>I am trying to build a Convolutional Neural Network and am facing a problem. The network has 3 Conv. layers, ReLU activated, pooled, batch normalized and then flattened. Here’s the model code:</p>
<pre><code class="lang-auto">in_features = 152 #in_features for Flatten(linear) layer calculated

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 2, stride=1, padding=1)
        self.bn1 = nn.BatchNorm2d(num_features=6)
        self.conv2 = nn.Conv2d(6, 8, 2, stride=1, padding=1)
        self.bn2 = nn.BatchNorm2d(num_features=8)
        self.conv3 = nn.Conv2d(8, 12, 2, stride=1, padding=1)
        self.bn3 = nn.BatchNorm2d(num_features=12)

        self.fc1   = nn.Linear(in_features, 36)
        self.fc2   = nn.Linear(36, 3)
        
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = F.max_pool2d(out, 2)
        
        out = F.relu(self.bn2(self.conv2(out)))
        out = F.max_pool2d(out, 2)
        
        out = F.relu(self.bn3(self.conv3(out)))
        out = F.max_pool2d(out, 2)
        
        out = out.view(out.size(0), -1)
        out = F.relu(self.fc1(out))
        out = F.softmax(self.fc2(out))
        return out

pytorch_model = Net()
</code></pre>
<p>I am trying to do multiclass classification and so using ‘Adam’ Optimizer and ‘CrossEntropyLoss’ function.</p>
<p>My input is of the size: (143256,1, 150, 3) and I am trying to train using the following code:</p>
<pre><code class="lang-auto">batch_size = 100
epochs = 10

pytorch_model = pytorch_model.double()

for epoch in range(epochs):
    for i in tqdm(range(0, len(X_TRAIN), batch_size)):
        batch_x = X_TRAIN[i:i+batch_size]
        batch_y = Y_TRAIN[i:i+batch_size]
        
        optimizer.zero_grad() # clears all the gradients
        
        outputs = pytorch_model(batch_x.double())
        loss = criterion(outputs, batch_y)
        loss.backward()
        optimizer.step()
    print(loss)
</code></pre>
<p>And when trying to train the model I get this error:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-189-e1854e4cd77a&gt; in &lt;module&gt;
---&gt; 15         outputs = pytorch_model(batch_x.double())
~/anaconda3/envs/LearningML/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
--&gt; 541             result = self.forward(*input, **kwargs)
&lt;ipython-input-184-d3ab227f7914&gt; in forward(self, x)
---&gt; 27         out = F.relu(self.fc1(out))
~/anaconda3/envs/LearningML/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
--&gt; 541             result = self.forward(*input, **kwargs)
~/anaconda3/envs/LearningML/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
---&gt; 87         return F.linear(input, self.weight, self.bias)
~/anaconda3/envs/LearningML/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
-&gt; 1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
   1372         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [100 x 228], m2: [152 x 36] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:197
</code></pre>
<p>I am fairly new to using PyTorch, so I know there’s something I am doing wrong here. Can anyone please explain what is that? Any help would be really appreciated…<br>
Thanks!</p>

username: ptrblck
cooked: <p>It seems you’ve defined <code>in_features</code> as 152, which does not match the flattened shape of the input tensor to <code>self.fc1</code>.<br>
If you add print statements right before the <code>self.fc1</code> call, you can simply check the shape, which will be <code>[batch_size, 228]</code>.<br>
If you create the first linear layer as:</p>
<pre><code class="lang-python">self.fc1 = nn.Linear(228, 36)
</code></pre>
<p>your code should work.</p>
<p>Also, <code>nn.CrossEntropyLoss</code> expects raw logits, so just remove the <code>softmax</code> at the end of your model and simply return the output of <code>self.fc2</code>.</p>

username: Eta_C
cooked: <p>Follow your code.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>layer</th>
<th>output_size</th>
</tr>
</thead>
<tbody>
<tr>
<td>input</td>
<td>(N, 1, 150, 3)</td>
</tr>
<tr>
<td>conv1_bn_relu</td>
<td>(N, 6, 151, 4)</td>
</tr>
<tr>
<td>max_pool2d</td>
<td>(N, 6, 75, 2)</td>
</tr>
<tr>
<td>conv2_bn_relu</td>
<td>(N, 8, 76, 3)</td>
</tr>
<tr>
<td>max_pool2d</td>
<td>(N, 8, 38, 1)</td>
</tr>
<tr>
<td>conv3_bn_relu</td>
<td>(N, 12, 39, 2)</td>
</tr>
<tr>
<td>max_pool2d</td>
<td>(N, 12, 19, 1)</td>
</tr>
<tr>
<td>view</td>
<td>(N, 228)</td>
</tr>
</tbody>
</table>
</div><p>So, <code>in_features</code> in <code>self.fc1</code> must be 228 instead of 152.</p>

username: Waqas
cooked: <p>Well, that really helped, and thanks for this…<br>
BUT, I guess there’s something else too that I’ve probably done wrong…</p>
<p>Now, it seems like it’s getting the outputs fine but has problem in finding loss and is expecting scalar type Long. Here’s the traceback for the RuntimeError:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-213-e1854e4cd77a&gt; in &lt;module&gt;
 ---&gt; 16         loss = criterion(outputs, batch_y)
 ~/anaconda3/envs/LearningML/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
 --&gt; 541             result = self.forward(*input, **kwargs)
~/anaconda3/envs/LearningML/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
--&gt; 916                                ignore_index=self.ignore_index, reduction=self.reduction)
~/anaconda3/envs/LearningML/lib/python3.6/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
-&gt; 2009     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
~/anaconda3/envs/LearningML/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-&gt; 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of scalar type Long but got scalar type Double for argument #2 'target' in call to _thnn_nll_loss_forward
</code></pre>

username: Waqas
cooked: <p>That is really cool!<br>
How can I print out all the layers like this?</p>

username: Eta_C
cooked: <p>I just… type them out manually…so that you can read more easily…<br>
But if you are interesting in how to print out the output shape of each layer automatically, here’s a simple tutorial about <a href="https://www.kaggle.com/sironghuang/understanding-pytorch-hooks" rel="nofollow noopener">forward hooks</a>.</p>

username: Waqas
cooked: <p>Oh right!<br>
Thank you for the tutorial on forward hooks as well… Really appreciated…! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: Waqas
cooked: <p>The error, I replied above with:</p>
<pre><code class="lang-auto">RuntimeError: Expected object of scalar type Long but got scalar type Double for argument #2 'target' in call to _thnn_nll_loss_forward
</code></pre>
<p>was resolved by converting both the arguments that loss criterion needed, outputs and batch_y, to torch type long tensors. By changing the code from:</p>
<pre><code class="lang-auto"> outputs = pytorch_model(batch_x.double())
 loss = criterion(outputs, batch_y)
</code></pre>
<p>to</p>
<pre><code class="lang-auto"> outputs = pytorch_model(batch_x.double()).long()
 loss = criterion(outputs, batch_y.long())
</code></pre>
<p>And now, it gives this error:</p>
<pre><code class="lang-auto">RuntimeError: "log_softmax_lastdim_kernel_impl" not implemented for 'Long'
</code></pre>
<p>How can I resolve this?</p>

username: Eta_C
cooked: <p><a href="https://pytorch.org/docs/stable/nn.html?highlight=cross#torch.nn.CrossEntropyLoss" rel="nofollow noopener">Here</a>’s the documentation about <code>CrossEntropyLoss</code>.</p>
<p><code>outputs</code>: shape (N, 3), type Float or Double or Half, <strong>without softmax</strong><br>
<code>batch_y</code>: shape (N,), type Long, the labels.</p>
<p><code>CrossEntropyLoss</code> equals to <code>LogSoftmax</code> + <code>NLLLoss</code></p>

username: Waqas
cooked: <p>Thank you so much <a class="mention" href="/u/eta_c">@Eta_C</a>!!<br>
It solved the problem… The model is now being trained…!</p>

username: Waqas
cooked: <p>Since my problem is resolved, all thanks to PyTorch’s amazing discussion community… For anyone who happens to read this post in future for the same problem, here’s my final working model class and the training code:</p>
<pre><code class="lang-auto">in_features = 228 #in_features needed for Flattening(linear/fully connected) layer

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 2, stride=1, padding=1)
        self.bn1 = nn.BatchNorm2d(num_features=6)
        self.conv2 = nn.Conv2d(6, 8, 2, stride=1, padding=1)
        self.bn2 = nn.BatchNorm2d(num_features=8)
        self.conv3 = nn.Conv2d(8, 12, 2, stride=1, padding=1)
        self.bn3 = nn.BatchNorm2d(num_features=12)

        self.fc1   = nn.Linear(in_features, 36)
        self.fc2   = nn.Linear(36, 3)
        
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = F.max_pool2d(out, 2)
        
        out = F.relu(self.bn2(self.conv2(out)))
        out = F.max_pool2d(out, 2)
        
        out = F.relu(self.bn3(self.conv3(out)))
        out = F.max_pool2d(out, 2)
        
        out = out.view(out.size(0), -1)
        print (out.size())  # helps in showing the size needed
        out = F.relu(self.fc1(out))
        out = self.fc2(out)
        return out

pytorch_model = Net()
</code></pre>
<p>Where the <code>in_features</code> can be calculated by following the code (e.g. in my case starting from <code>input size &gt; conv1 (bn1, relu) &gt; max_pool2d &gt; conv2 (bn2, relu)</code> etc…). and using the following formula to find output size for each layer and finally the output size that <code>nn.Linear</code> layer needs (which led me to 228 (12*19)):</p>
<blockquote>
<p>O=( (W−K+2P) /S ) + 1<br>
where<br>
W=Input height/width<br>
K=Filter size<br>
P=Padding<br>
S=Stride<br>
O=Output size</p>
</blockquote>
<p>The criterion and optimizer:</p>
<pre><code class="lang-auto">weights = torch.tensor([0.05, 0.15, 0.8]).double() # assigning weights to classes
criterion = nn.CrossEntropyLoss(weight=weights)
optimizer = optim.Adam(pytorch_model.parameters())
</code></pre>
<p>And finally to train the model:</p>
<pre><code class="lang-auto">batch_size = 100
epochs = 10

# Uncomment the line below if data sets are type double. 
# Or can conversely convert data sets to type float and leave this commented

# pytorch_model = pytorch_model.double()

for epoch in tqdm(range(epochs)):
    for i in range(0, len(X_TRAIN), batch_size):
        batch_x = X_TRAIN[i:i+batch_size]
        batch_y = Y_TRAIN[i:i+batch_size]
        
        optimizer.zero_grad() # clears all the gradients
        
        outputs = pytorch_model(batch_x.double())
        loss = criterion(outputs, batch_y.long())
        loss.backward()
        optimizer.step()
    print(loss)
</code></pre>
<p>Making sure to follow the suggestions from <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/eta_c">@Eta_C</a>, I managed to run the training successfully.<br>
Happy Learning!</p>

username: Eta_C
cooked: <p>Emmmm, in most cases, <code>Float</code> is enough…<br>
<code>pytorch_model.float()</code> and <code>batch_x.float()</code>…<br>
<code>Double</code> would use more memory and make training slower…</p>

username: Waqas
cooked: <p>That was exactly what was happening!<br>
It <strong>is</strong> utilizing a lot of memory and training a lot slower than usual. Even 10 epochs are taking about 50 minutes. Thanks for pointing this out…!</p>