Concatenating flattened tensor in CNN with another tensor

Antonio_Linares: <p>Hi all,</p>
<p>Following with my <a href="https://discuss.pytorch.org/t/looping-over-2-dataloaders-simultaneously-one-for-images-another-for-numbers/81207/3">last post</a>, I am now trying to concatenate the flattened output of a CNN with another tensor, in the forward pass through the network.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a71ab512b88e3fc8ec86090ba35acb27c14a659e.png" data-download-href="https://discuss.pytorch.org/uploads/default/a71ab512b88e3fc8ec86090ba35acb27c14a659e" title="cnn"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a71ab512b88e3fc8ec86090ba35acb27c14a659e_2_690x288.png" alt="cnn" data-base62-sha1="nQgXnvioJ4XCdDw5yltxqGaSabs" width="690" height="288" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a71ab512b88e3fc8ec86090ba35acb27c14a659e_2_690x288.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a71ab512b88e3fc8ec86090ba35acb27c14a659e_2_1035x432.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a71ab512b88e3fc8ec86090ba35acb27c14a659e_2_1380x576.png 2x" data-dominant-color="E4E7ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cnn</span><span class="informations">1512×633 34.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The pink area is the tensor that is to be concatenated to the flattened output.</p>
<p>My CNN is defined as such:</p>
<pre><code class="lang-auto">#defining the model with an object oriented approach
from torch import nn, cuda
from torch.nn import functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
      
        #defining convolutional kernels
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5, stride=1, padding=2)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=3, stride=1, padding=1)
        
        #defining linear layer
        self.fc1 = nn.Linear(259, 2)

        #defining maxpool
        self.pool = nn.MaxPool2d(2, 2)
        
        #defining activation function
        self.leaky = nn.LeakyReLU()
    
    def forward(self, x, attributes):
        #passing through conv1
        x = self.pool(self.leaky(self.conv1(x)))
        #passing through conv2
        x = self.pool(self.leaky(self.conv2(x)))
        
        #flattening the low level representation of the image
        x = x.view(x.size(0), -1)
        
        #merging resulting flattened vector with incoming attributes vector
        x = torch.cat((x, attributes), 1)
        
        #passing (batch_size, 259) through linear layer and logsoftmax (to obtain probability scores)
        x = nn.LofSoftmax(self.fc1(x))
        
        return x
</code></pre>
<p>My training loop so far is defined as follows:</p>
<pre><code class="lang-auto">#training loop
for i in range(500): 
    #iterating through data, one batch at a time, for the 2 data loaders
    model.train()
    train_loss = 0.0
    valid_loss = 0.0
    for b, b_2 in zip(mini_b, mini_b2):
        #unpacking image (x) and label(y) from b
        b_x, b_y = b
        #element 0 of b_2 is the attributes associated to b_x
        b_2 = b_2[0]
        print(b_2.shape)
        
        optimizer.zero_grad()
        output = model(b_x, b_2)
        loss = criterion(output, b_y)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()
</code></pre>
<p>Where mini_b and mini_b2 are 2 data loader objects of the same length. The pink tensor comes from mini_b2 (b_2 in the loop above).</p>
<p>Getting the following error:</p>
<p>TypeError: conv2d(): argument ‘input’ (position 1) must be Tensor, not module</p>

Isaac_Kargar: <p>What is the type of <code>b_x</code> and <code>b_2</code> in your code? I\m trying to run your code but I cannot. Is it possible to put a sample code that I can run it?</p>

Antonio_Linares: <p>Hey <a class="mention" href="/u/isaac_kargar">@Isaac_Kargar</a>,</p>
<p>I fixed this by converting both datasets to the same tensor type (both code snippets in the original post unchanged), as follows:</p>
<pre><code class="lang-auto">#converting datasets to tensors
ext_data_tensor_train = torch.tensor(ext_data_train.values, requires_grad=False, dtype=torch.float)
ext_data_tensor_test = torch.tensor(ext_data_test.values, requires_grad=False, dtype=torch.float)

#batching data accordingly
#batching training data
mini_b2 = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(ext_data_tensor_train), 
                                       batch_size=100, shuffle=False) 
#batching testing data
mini_b2_test = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(ext_data_tensor_test), 
                                       batch_size=100, shuffle=False)
</code></pre>
<p>Looking at the definition of the Net class, in the forward pass:</p>
<p>Before the concat operation:</p>
<p>x.shape is [batch_size, 240]<br>
attributes.shape is [batch_size, 19]</p>
<p>After the concat operation:</p>
<pre><code class="lang-auto">x = torch.cat((x, attributes), 1)
</code></pre>
<p>x.shape is now [batch_size, 240 + 19]</p>