Concatenating image tensor and numerical data not matching up

username: Jordan_Howell
cooked: <p>Hello,</p>
<p>I’m trying to slowly build an image model that is concatenated with  numerical data and categorical embedding layers.  That said, I’m only on the <code>image+numerical data</code> step.  Following this post: <a href="https://discuss.pytorch.org/t/concatenate-layer-output-with-additional-input-data/20462" class="inline-onebox">Concatenate layer output with additional input data</a>, I’m getting an error in the dimensions.</p>
<p>Here is the error traceback:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-255-8c805dc65c00&gt; in &lt;module&gt;
     20 
     21 
---&gt; 22         y_pred = combined_model(image, numerical_data)
     23         single_loss = loss_function(y_pred, label)
     24         aggregated_losses.append(single_loss)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--&gt; 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

&lt;ipython-input-250-945f067aedac&gt; in forward(self, image, x_numerical)
     26         x1 = self.cnn(image)
     27         x2 = numerical_data
---&gt; 28         x = torch.cat((x1, x2), dim = 1)
     29         x = F.relu(self.fc1(x))
     30         x = self.fc2(x)

RuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 3 and 2 at C:/w/1/s/windows/pytorch/aten/src\THC/generic/THCTensorMath.cu:62
</code></pre>
<p>Here is my model object:</p>
<pre><code class="lang-auto">class Image_Embedd(nn.Module):

    def __init__(self):
        '''
        Args
        ---------------------------
        embedding_size: Contains the embedding size for the categorical columns
        num_numerical_cols: Stores the total number of numerical columns
        output_size: The size of the output layer or the number of possible outputs.
        layers: List which contains number of neurons for all the layers.
        p: Dropout with the default value of 0.5
        
        '''
        super(Image_Embedd, self).__init__()
        
        self.cnn = models.resnet50(pretrained=False)
        
        self.cnn.fc = nn.Linear(self.cnn.fc.in_features, 256)
        self.cnn.fc1 = nn.Linear(256 + numerical_data.shape, 256 + numerical_data.shape)
        self.cnn.fc2 = nn.Linear(256 + numerical_data.shape, 2)
        
        
    #define the foward method
    def forward(self, image, x_numerical):
        
        x1 = self.cnn(image)
        x2 = numerical_data
        x = torch.cat((x1, x2), dim = 1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = F.log_softmax(x)
        return x
</code></pre>
<p>Here is my model run:</p>
<pre><code class="lang-auto">epochs = 1
aggregated_losses = []

max_trn_batch = 25

for i in range(epochs):
    for b, (image, label, policy, cateogrical_data, numerical_data
            , categorical_embedding_sizes) in enumerate(train_loader):
        image = image.cuda()
        label = label.cuda()
        numerical_data = numerical_data.cuda()
        #print(image, label, categorical_data, numerical_data)
        
        #count batches
        b += 1
        
        #throttle teh batches
        if b == max_trn_batch:
            break
        

        y_pred = combined_model(image, numerical_data)
        single_loss = loss_function(y_pred, label)
        aggregated_losses.append(single_loss)
        
        # statistics
        running_loss += single_loss.item() * image.size(0)
        running_corrects += torch.sum(y_pred == label.data)
        


        print(f'train-epoch: {i}, train-batch: {b}')

        optimizer.zero_grad()
        single_loss.backward()
        optimizer.step()
</code></pre>
<p>When I run:</p>
<pre><code class="lang-auto">for image, label, policy, cateogrical_data, numerical_data, categorical_embedding_sizes in train_loader: 
    print(f"numeric size is {numerical_data.shape} \
          image size is {image.shape}")
    break
</code></pre>
<p>I get:</p>
<blockquote>
<p>numeric size is torch.Size([10, 110528, 8])           image size is torch.Size([10, 3, 224, 224])</p>
</blockquote>
<p>I’m not sure how to properly concat those two sizes.</p>
<p>If there is anything else that will help, or if you see something glaring that I’m missing, I would appreciate the help.  Thank you.</p>

username: ptrblck
cooked: <p>The output of the resnet will be <code>[batch_size, 1000]</code>, while <code>x_numerical</code> will be <code>[batch_size, 110528, 8]</code>, which is incompatible, since these tensors do not have the same number of dimensions.<br>
How would you like to concatenate the 2-dimensional cnn output with the 3-dimensional numerical tensor?</p>

username: Jordan_Howell
cooked: <p>Thank you for the reply.</p>
<p>For the numerical data, I figured out it was in the data loader/custom data set.  I was  pulling the every numerical value for every observation.  By changing<br>
<code>numerical_data = self.image_frame.loc[numerical_columns]</code> to <code>numerical_data = self.image_frame.loc[idx, numerical_columns]</code>, it now concatenates just fine now and runs.</p>
<p>Now i need to figure out how to properly add categorical embeddings in the model object.</p>