Passing a vector alongside sequences in a batch

Mathys_Gr: <p>Hi all, I hope you are well,</p>
<p>I have a problem where I have a CNN with two inputs, one is a 75 x 4 matrix and the other one is a single value. With a batch size of 512, the size of the training batch matrix is naturally 512 x 75 x 4, and the network trains fine without the second input.<br>
But, as soon as I try to add the vector of single values, each 1 x 75 x 4 Matrix is passed alongside the whole vector (512 entries) which raises an error.</p>
<p>During the training, I would like to forward pass each 1 x 75 x 4 matrix alongside its corresponding single value but I’ve yet to find how,</p>
<p>Thanks in advance,</p>
<p>Mat</p>

ptrblck: <p>Could you post the error message and a small code snippet so that we could have a look?</p>

Mathys_Gr: <p>Hi, thank you for your response, sure :</p>
<p>Here is my collate function which retrieves the length of one-hot-encoded sequences lengths, and then returns padded sequence, and lengths. (tag corresponds to the y, this is a regression problem).</p>
<pre><code class="lang-auto">def collate(batch):

    ## Get sequence lengths 
    lengths = torch.IntTensor([np.array(t['seq']).shape[0] for t in batch  ])
    ## padd
    batch_ = [torch.Tensor(t['seq']).to(device) for t in batch]
    batch_ = torch.nn.utils.rnn.pad_sequence(batch_, batch_first=True)
    #batch_ = torch.nn.utils.rnn.pack_padded_sequence(batch_, batch_first=True)
    
    # Get tags
    tags = [t['tag'] for t in batch]
    tags = torch.FloatTensor(tags)
    
    return batch_, lengths, tags
</code></pre>
<p>Here is a snippet of my forward function in my CNN :</p>
<pre><code class="lang-auto">    def forward(self, x, length):
        
        # Would like to resize x based on the length here :
        # Something like x = x[:length,:]

        
        x = self.conv1(x)
        x = F.relu(x)
        x = self.batch1(x)
</code></pre>
<p>And this is my training loop :</p>
<pre><code class="lang-auto">    for i_batch, item in enumerate(train_loader):
              
        seqs = item[0]
        tags = item[2]
        lengths = item[1]
        # (1) :
        outputs = custom_model(seqs, lengths)  
</code></pre>
<p>the line after #(1): raises an error because lengths is a vector. But I would like to apply the resize function to each of my sequences based on the corresponding value in the length vector.</p>
<p>If you would have more questions, don’t hesitate !</p>
<p>Mat</p>

ptrblck: <p>Thanks for the code! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Could you please also post the error message?</p>

Mathys_Gr: <p>Welcome !<br>
Instead of the <span class="hashtag">#resize</span> line, I’ve used :</p>
<pre><code class="lang-auto">def forward(self, x, length):
      x = torch.narrow(0,0,length) #this line instead of the resize line in the previous forward code
      x = self.conv1(x)
</code></pre>
<p>And it raises the error : narrow(): argument ‘length’ (position 4) must be int, not Tensor</p>
<p>Further analysis showed that ‘length’ is my 512 x 1 tensor with the 512 different lengths (512 being the batch size).<br>
I would like to apply the narrow function to each sequence, so that each sequence is “narrowed” by it’s corresponding length in the length vector.</p>
<p>Thanks for your time,</p>

ptrblck: <p>Unfortunately this won’t work. as <code>x</code> would contain tensors with different lengths, if I understand the use case correctly.<br>
Could you zero-pad <code>x</code> and use <code>lengths</code> as a mask?<br>
If not, could you explain your use case a bit, please?</p>

Mathys_Gr: <p>Hi again !<br>
Here is an explanation of my use-case :</p>
<p>I have one-hot encoded sequences which are L x 4 in dimensions (L being the lenght of the sequence).<br>
They look like this :<br>
[[1,0,0,0],<br>
[1,0,0,0],<br>
[1,0,0,0],<br>
[0,1,0,0],<br>
[0,0,1,0],<br>
[1,0,0,0],<br>
[0,0,0,1],<br>
[1,0,0,0]] (just a silly example).</p>
<p>These sequences are associated with a value that I am trying to predict.</p>
<p>They have various sizes, and I’ve tried 0 padding them before (which works) but I am afraid this will lead to some problems because we are looking at aligned motifs. (example a [1,0,0,0] followed by a [0,0,1,0] that are always at the same position). I am afraid that padding would remove this information.<br>
As for a mask, I’ve not thought of this, do you have any general idea how it could be implemented so that the network “cuts” the padded part of the sequences ?</p>
<p>Edit : For now I’ve gone around the problem by using a batch_size of one, but it greatly increases computing time, which is going to be a burden when I try to scale this up to datasets of over 1,2 or 3M sequences.</p>
<p>Thank you very much for your answers,</p>

ptrblck: <p>What would be the desired shape of <code>x</code> after the <code>narrow</code> operation?<br>
Would it be in the desired use case <code>[batch_size, 4, varying_shape]</code>?</p>

Mathys_Gr: <p>For now, my model works with tensor of shapes : [batch_size, 1, varying_shape, 4]</p>