Variable length sequences and pack_padded_sequence

shayben: <ol>
<li>I see &gt;5x slowdown on the .backward() call when using pack_padded_sequence (~80s instead of ~14s). Is this expected?</li>
<li>I am currently using a stack of 2 bidirectional lstms, what is the best practice for retrieving the final states when running on variable length inputs? I did this (not sure if correct / fastest approach):</li>
</ol>
<pre><code class="lang-auto">rnn_out, (ht, ct) = lstm_layer(lstm_input) 
reshaped_hidden = ht.view(num_lstm_layers, 2, batch_size, hidden_size)
back_forward_concat = torch.cat([reshaped_hidden[-1, 0, :, :], reshaped_hidden[-1, 1, :, :]], dim=1)
</code></pre>
<p>Thanks!</p>

shayben: <p>I ran torch.utils.bottleneck to try and identify the cause, but I am not sure if this is informative enough:</p>
<p>Without pack_padded_sequence:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5e25055dc95c0e5a410f37f39c50d95ed012775.png" data-download-href="https://discuss.pytorch.org/uploads/default/c5e25055dc95c0e5a410f37f39c50d95ed012775" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5e25055dc95c0e5a410f37f39c50d95ed012775.png" alt="image" data-base62-sha1="seyW4Z17cELE6nuHpujlCUFNoxf" width="582" height="500" data-dominant-color="050505"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1149×987 24.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With pack_padded_sequence:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbedf9cfcbe63d7c8f8be0c9267d28453c856f39.png" data-download-href="https://discuss.pytorch.org/uploads/default/fbedf9cfcbe63d7c8f8be0c9267d28453c856f39" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbedf9cfcbe63d7c8f8be0c9267d28453c856f39.png" alt="image" data-base62-sha1="zWFHLDoJEpa6OoZtLxXzzGbSHDr" width="690" height="483" data-dominant-color="070707"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1114×781 23.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

richard: <p>Could you post code snippets pertaining to each of the two runs? It is hard to tell what is going on without them.</p>

shayben: <pre><code class="lang-auto">        concatenated = torch.cat(transformed, dim=2)
        if True:
            packed = pack_padded_sequence(concatenated, input_lengths, batch_first=True, enforce_sorted=False)
            rnnout, _ = self.core_layers.lstm(packed)  # out: tensor of shape (batch_size, seq_length, hidden_size*2)
            unpacked = pad_packed_sequence(rnnout, batch_first=True)
            final_per_seq = unpacked[0][torch.arange(unpacked[0].size(0)), unpacked[1]-1]
            out = self.core_layers.fc(final_per_seq)
        else: # I am using this to debug runtime issues of padding
            concat_normed = self.core_layers.layernorm(concatenated)
            rnnout, _ = self.core_layers.lstm(concat_normed)  # out: tensor of shape (batch_size, seq_length, hidden_size*2)
            out = self.core_layers.fc(rnnout[:, -1, :])  # this is probably the wrong way to index variable length sequences.
</code></pre>

shayben: <p>I have verified that this problem does not reproduce on GPU, this is a sufficient workaround for me.<br>
Thanks.</p>

richard: <p>PackedSequence is definitely more efficient on CUDA because pytorch calls into a special cudnn kernel to perform that computation.</p>