Trouble with packed_padded_sequence and LSTM for variable length inputs

username: markolonius
cooked: <p>Its been months I’ve been trying to use pack_padded_sequence with LSTM.  My current setup I’m working with data that is in a python list of tensors shape 2x(some variable length) such as torch.Size([2, 2466]).</p>
<p>I have a data loader with a custom collate_fn that is pretty much same as found here: <a href="https://www.codefull.net/2018/11/use-pytorchs-dataloader-with-variable-length-sequences-for-lstm-gru/" rel="nofollow noopener"> Use PyTorch’s DataLoader with Variable Length Sequences for LSTM/GRU</a> with the exception I don’t sort in order of length.  I’m using pytorch 1.1.0.</p>
<pre><code class="lang-auto">def padSequence(batch):
  # each element in "batch" is a tuple (data, label)
  # Get each sequence and pad it
  sequences = [x[0] for x in batch] #sorted_batch]
  sequences_padded = torch.nn.utils.rnn.pad_sequence(sequences, batch_first=True) 
  # Also need to store the length of each sequence
  # This is later needed in order to unpad the sequences
  seq_lengths = [len(x) for x in sequences]
  # need to pad labels too
  labels = [x[1] for x in batch]
  labels_padded = torch.nn.utils.rnn.pad_sequence(labels,batch_first=True)
  label_lengths = [len(x) for x in labels] 
  return sequences_padded, seq_lengths, labels_padded, label_lengths
</code></pre>
<p>The data loader’s output with batch=5 gives:</p>
<pre><code class="lang-auto">tensor([[8.2430, 8.2793, 8.3186,  ..., 0.0000, 0.0000, 0.0000],
        [6.6331, 6.6288, 6.6292,  ..., 0.0000, 0.0000, 0.0000],
        [4.2062, 4.2408, 4.2675,  ..., 3.4694, 3.4807, 3.4933],
        [3.5047, 3.5154, 3.5313,  ..., 0.0000, 0.0000, 0.0000],
        [5.4685, 5.4138, 5.3533,  ..., 0.0000, 0.0000, 0.0000]],
       dtype=torch.float64)
</code></pre>
<p>And a list of sequence lengths: <code>[474, 473, 1160, 533, 555]</code></p>
<p>Feeding it to <code>torch.nn.utils.rnn.pack_padded_sequence(sequences_padded, seq_lengths, batch_first=True, enforce_sorted=False)</code> I get an output something like this:</p>
<pre><code class="lang-auto">PackedSequence(data=tensor([4.2062, 5.4685, 3.5047,  ..., 3.4694, 3.4807, 3.4933],
       dtype=torch.float64), batch_sizes=tensor([5, 5, 5,  ..., 1, 1, 1]), sorted_indices=tensor([2, 4, 3, 0, 1]), unsorted_indices=tensor([3, 4, 0, 2, 1]))
</code></pre>
<p>And then passing to my lstm I get the error:<br>
<code>RuntimeError: input must have 2 dimensions, got 1</code></p>
<p>Which I found similar in this post: <a href="https://stackoverflow.com/questions/53197659/pytorch-passing-packsequence-argument-to-lstm" rel="nofollow noopener">Pytorch passing PackSequence argument to LSTM</a></p>
<p>Now I tried making my data in the same shape with my collate_fn:</p>
<pre><code class="lang-auto">def padSequence(batch):
  sequences = [x[0].reshape((x[0].shape[0],1)) for x in batch]
  sequences_padded = torch.nn.utils.rnn.pad_sequence(sequences, batch_first=True)
  seq_lengths = [len(x) for x in sequences] 
  labels = [x[1].reshape((x[1].shape[0],1)) for x in batch]
  labels_padded = torch.nn.utils.rnn.pad_sequence(labels,batch_first=True)
  label_lengths = [len(x) for x in labels] #torch.LongTensor([len(x) for x in labels])
  return sequences_padded, seq_lengths, labels_padded, label_lengths
</code></pre>
<p>However then I keep getting the error: <code>RuntimeError: input.size(-1) must be equal to input_size. Expected 2, got 1</code></p>
<p>I’ve tried changing the shape of the original data making it a list of tensors with variable length x 2 and so many other things getting same errors.  At this point I don’t know what is wrong and where to go from here in understanding what pytorch input wants for a simple LSTM w/ packed_padded sequence…  I’ve read through so many posts and searched a lot these past few months with no help at all.  Any direction would help.</p>
<p>Including this if could give any clues:</p>
<pre><code class="lang-auto">input_size = 1 
hidden_size = 1 
output_size = 1 
num_layers = 2
num_classes = 4

class Model(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(Model, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional=False)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x, X_lengths):
      
      # Set initial hidden and cell states 
      h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).cuda()
      c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).cuda()
      
      packed = torch.nn.utils.rnn.pack_padded_sequence(x, X_lengths, batch_first=True, enforce_sorted=False)
      # Forward propagate LSTM
      X, _ = self.lstm((packed), (h0,c0))
      #X, _ = torch.nn.utils.rnn.pad_packed_sequence(X, batch_first=True)

      out = self.fc(X) #[:, -1, :])
      return nn.functional.softmax(out,dim=2)
</code></pre>
<p>Thanks!</p>

username: markolonius
cooked: <p>Turns out I didn’t know how to assign variables to cuda or cpu and things got all out of whack.  Turns out my packed padded sequence was not assigned to a cuda device.  Here is a simple example I used to test things that helped me figure it out.</p>
<pre><code class="lang-auto">import torch 
import torch.nn.utils.rnn as rnn_utils 
a = torch.Tensor([[1], [2], [3]]) 
b = torch.Tensor([[4], [5]]) 
c = torch.Tensor([[6]]) 
d = torch.Tensor([[7],[8],[9],[10]])
batch = [a,b,c,d]
padded= rnn_utils.pad_sequence(batch, batch_first=True)
sorted_batch_lengths = [len(x) for x in padded]
packed = rnn_utils.pack_padded_sequence(padded, sorted_batch_lengths, batch_first=True, enforce_sorted=False).cuda()
lstm = torch.nn.LSTM(input_size=1, hidden_size=3, batch_first=True).cuda()
lstm(packed)
</code></pre>
<p>Hopefully this will help someone else out in the future so you don’t spend months stuck.</p>