Getting a size mismatch error that I'm missing in the code:

Jordan_Howell: <p>Hello. I’m getting a size mismatch error in a model where i’m combining the output from resnet-50 with tabular data.  However, I can’t seem to figure out how to fix it.  I don’t seem to understand the correct way to calc these layers.</p>
<p>Model below:</p>
<pre><code class="lang-auto">class Image_Embedd(nn.Module):

    def __init__(self, embedding_size):
        '''
        Args
        ---------------------------
        embedding_size: Contains the embedding size for the categorical columns
        num_numerical_cols: Stores the total number of numerical columns
        output_size: The size of the output layer or the number of possible outputs.
        layers: List which contains number of neurons for all the layers.
        p: Dropout with the default value of 0.5
        
        '''
        super().__init__()    
        
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
        self.embedding_dropout = nn.Dropout(p = .04)
        
        self.cnn = models.resnet50(pretrained=False).cuda()
        
        self.cnn.fc = nn.Linear(self.cnn.fc.in_features, 1000)
        self.fc1 = nn.Linear(1000, 1077)
        self.fc2 = nn.BatchNorm1d(1077)
        self.fc3 = nn.Dropout(p = .04)
        self.fc4= nn.Linear(1077, 256)
        self.fc5= nn.BatchNorm1d(256)
        self.fc6= nn.Dropout(p = .04)  
        self.fc7= nn.Linear(256, 128)
        self.fc8= nn.BatchNorm1d(128)
        self.fc9= nn.Dropout(p = .04)                            
        self.fc10= nn.Linear(128, 32)
        self.fc11= nn.BatchNorm1d(32)
        self.fc12= nn.Dropout(p = .04)
        self.fc13= nn.Linear(32, 2)       
        
    #define the foward method
    def forward(self, image, x_numerical, x_categorical):
        
        embeddings = []
        for i, e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
            
        x = torch.cat(embeddings, 1)
        x = self.embedding_dropout(x)
        x1 = self.cnn(image)
        x2 = x_numerical
        
        x3 = torch.cat((x1, x2), dim = 1)
        x4 = torch.cat((x, x3), dim = 1)
        
        x4 = F.relu(self.fc1(x4))
        x4 = self.fc2(x4)
        x4 = self.fc3(x4)
        x4 = F.relu(self.fc4(x4))
        x4 = self.fc5(x4)
        x4 = self.fc6(x4)
        x4 = F.relu(self.fc7(x4))
        x4 = self.fc8(x4)
        x4 = self.fc9(x4)
        x4 = F.relu(self.fc10(x4))
        x4 = self.fc11(x4)
        x4 = self.fc12(x4)
        x4 = self.fc13(x4)
        x4 = F.log_softmax(x4)
        return x4
</code></pre>
<p>Here is the error traceback:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-111-780fd4df5eca&gt; in &lt;module&gt;
     18             break
     19 
---&gt; 20         y_pred = combined_model(image, numerical_data, categorical_data)
     21         single_loss = criterion(y_pred, label)
     22 

~\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

&lt;ipython-input-108-0bf79a481817&gt; in forward(self, image, x_numerical, x_categorical)
     49         x4 = torch.cat((x, x3), dim = 1)
     50 
---&gt; 51         x4 = F.relu(self.fc1(x4))
     52         x4 = self.fc2(x4)
     53         x4 = self.fc3(x4)

~\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     85 
     86     def forward(self, input):
---&gt; 87         return F.linear(input, self.weight, self.bias)
     88 
     89     def extra_repr(self):

~\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
-&gt; 1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
   1372         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [10 x 1077], m2: [1000 x 1077] at C:/w/1/s/tmp_conda_3.7_100118/conda/conda-bld/pytorch_1579082551706/work/aten/src\THC/generic/THCTensorMathBlas.cu:290
</code></pre>
<p>Seems to be throwing at <code>fc1</code></p>

LeviViana: <p>Try putting some size prints before the error. I’d try:</p>
<aside class="quote no-group quote-modified" data-username="Jordan_Howell" data-post="1" data-topic="67552">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jordan_howell/48/16421_2.png" class="avatar"> Jordan_Howell:</div>
<blockquote>
<pre><code class="lang-auto">x1 = self.cnn(image)
print("x1", x1.size()) # &lt;- here
x2 = x_numerical
print("x2", x2.size()) # &lt;- here
x3 = torch.cat((x1, x2), dim = 1)
print("x3", x3.size()) # &lt;- here
x4 = torch.cat((x, x3), dim = 1)
print("x4", x4.size()) # &lt;- here
x4 = F.relu(self.fc1(x4))
</code></pre>
</blockquote>
</aside>
<p>Maybe just putting <code>self.fc1 = nn.Linear(1077, 1077)</code> will fix it, but I don’t know if is this what you want there.</p>

Jordan_Howell: <p>Thank you @ LevinViana.<br>
Here is my print statements:</p>
<pre><code class="lang-auto"> x = torch.cat(embeddings, 1)
        print("cat", x.size())
        x = self.embedding_dropout(x)
        x1 = self.cnn(image)
        print("image", x1.size())
        x2 = x_numerical
        print("numerical", x2.size())
        
        x3 = torch.cat((x1, x2), dim = 1)
        print("image+numerical", x3.size())
        x4 = torch.cat((x, x3), dim = 1)
        print("cat+image_numerical", x4.size())
        
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-auto">cat torch.Size([10, 69])
image torch.Size([10, 1000])
numerical torch.Size([10, 8])
image+numerical torch.Size([10, 1008])
cat+image_numerical torch.Size([10, 1077])
</code></pre>
<p>I don’t understand why the shapes are as they are.  If the cnn’s final layer is 1000 and I concat 77 features from the categorical and numerical features, why it errors out.</p>
<p>As an aside, the <code>(1077, 1077)</code> works.  Can you explain why it works for my knowledge?</p>