Why do we need to pack padded batches of sequences in Pytorch?

Brando_Miranda: <p>I was going through the <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="noopener nofollow ugc">Chatbot tutorial</a> and saw the following code:</p>
<pre><code class="lang-auto">    def forward(self, input_seq, input_lengths, hidden=None):
        '''
        input_seq : vector of indices of the words (max_length, batch_size)

        Computation Graph:

            Convert word indexes to embeddings.
            Pack padded batch of sequences for RNN module.
            Forward pass through GRU.
            Unpack padding.
            Sum bidirectional GRU outputs.
            Return output and final hidden state.
        '''
        # Convert word indexes (10,64) to embeddings (10, 64, 500)
        embedded = self.embedding(input_seq) # (10,64,500) = (max_len, batch_size, embedding_dim=hidden_dim)
        # Pack padded batch of sequences for RNN module
        packed = nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)
        # Forward pass through GRU
        outputs, hidden = self.gru(packed, hidden)
        # Unpack padding
        outputs, _ = nn.utils.rnn.pad_packed_sequence(outputs)
        # Sum bidirectional GRU outputs
        outputs = outputs[:, :, :self.hidden_size] + outputs[:, : ,self.hidden_size:]
        # Return output and final hidden state
        return outputs, hidden
</code></pre>
<p>I don’t understand what the purpose of packing is. I assume is that I first don’t understand the problem these functions are trying to solve so I am going to state what I think it is and hopefully this can help people understand whats going and correct it:</p>
<blockquote>
<p>Problem statement: process batches of different/variable length in Pytorch efficiently (rather than manually)</p>
</blockquote>
<p>This is my understanding of what is going on and how the functions solve it: We have to process batches of DIFFERENT lengths but we don’t want the padding to be processed by our RNN modules. So we need to tell our RNN what the padding is to process it properly. This is better than processing each sequence in the batch ourselves in pytorch by looping through it. My guess is because pytorch might to some optimization (with GPUs especially) than our code (in Python) sequentially going through each sequence in the loop. Adding parallelize ourselves seems silly.</p>
<p>So we pack the (zero) padded sequence and the packing tells pytorch how to have each sequence when the RNN model (say a GRU or LSTM) receives the batch so that it doesn’t process the meaningless padding (since the padding is only there so that things are tensors, since we can’t have “tensors of each row having a different length”)</p>
<p>Is this correct? Is this why we need padding?</p>
<hr>
<p>Crossposted:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.quora.com/unanswered/Why-do-we-need-to-pack-pad-batches-of-sequences-in-Pytorch">
  <header class="source">
      <img src="https://qsf.cf2.quoracdn.net/-4-images.favicon-new.ico-26-07ecf7cd341b6919.ico" class="site-icon" width="" height="">

      <a href="https://www.quora.com/unanswered/Why-do-we-need-to-pack-pad-batches-of-sequences-in-Pytorch" target="_blank" rel="noopener nofollow ugc">Quora</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://qsf.cf2.quoracdn.net/-4-images.Illustration_fb_share_default_1280x720.png-26-7b8bc71e8107b6f5.png" class="thumbnail">

<h3><a href="https://www.quora.com/unanswered/Why-do-we-need-to-pack-pad-batches-of-sequences-in-Pytorch" target="_blank" rel="noopener nofollow ugc">Why do we need to pack pad batches of sequences in Pytorch?</a></h3>

  <p>1 person wants answers to this question. Be the first to answer.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<hr>
<p>Resources I’ve read to understand this issue:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.quora.com/What-does-packing-a-sequence-mean-in-Pytorch">
  <header class="source">
      <img src="https://qsf.cf2.quoracdn.net/-4-images.favicon-new.ico-26-07ecf7cd341b6919.ico" class="site-icon" width="" height="">

      <a href="https://www.quora.com/What-does-packing-a-sequence-mean-in-Pytorch" target="_blank" rel="noopener nofollow ugc">Quora</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://qsf.cf2.quoracdn.net/-4-images.Illustration_fb_share_default_1280x720.png-26-7b8bc71e8107b6f5.png" class="thumbnail">

<h3><a href="https://www.quora.com/What-does-packing-a-sequence-mean-in-Pytorch" target="_blank" rel="noopener nofollow ugc">What does packing a sequence mean in Pytorch?</a></h3>

  <p>Answer: Generally - matrises and Tensors are like Multi-Dimensional Arrays with Depth.

This can be represented in different ways - Tuples (Lists consisting of Lists) and other dynamics.

To cohesively retain size - we tend to pad to not violate...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.quora.com/Why-do-we-need-to-pad-sequences-by-ourselves-in-Pytorch-when-we-supply-the-sequence-lengths-Why-isn-t-pad_packed_sequence-sufficient">
  <header class="source">
      <img src="https://qsf.cf2.quoracdn.net/-4-images.favicon-new.ico-26-07ecf7cd341b6919.ico" class="site-icon" width="" height="">

      <a href="https://www.quora.com/Why-do-we-need-to-pad-sequences-by-ourselves-in-Pytorch-when-we-supply-the-sequence-lengths-Why-isn-t-pad_packed_sequence-sufficient" target="_blank" rel="noopener nofollow ugc">Quora</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://qsf.cf2.quoracdn.net/-4-images.Illustration_fb_share_default_1280x720.png-26-7b8bc71e8107b6f5.png" class="thumbnail">

<h3><a href="https://www.quora.com/Why-do-we-need-to-pad-sequences-by-ourselves-in-Pytorch-when-we-supply-the-sequence-lengths-Why-isn-t-pad_packed_sequence-sufficient" target="_blank" rel="noopener nofollow ugc">Why do we need to pad sequences by ourselves in Pytorch when we supply the...</a></h3>

  <p>Answer: Due to it defaulting to padding up to the longest sequence that it can effectively see.

If a extended length is found, in the total pooled version - it’ll be incompatible in Lengths.

So you standardize to fit your longest sequence - and...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="quote quote-modified" data-post="1" data-topic="4099">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vijendra_rana/48/786_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/understanding-pack-padded-sequence-and-pad-packed-sequence/4099">Understanding pack_padded_sequence and pad_packed_sequence</a> 
  </div>
  <blockquote>
    Hi, 
I have a problem understanding these 2 utilities. Not able to figure out what it does. 
For eg. I was trying to replicate this with example from <a href="https://discuss.pytorch.org/t/simple-working-example-how-to-use-packing-for-variable-length-sequence-inputs-for-rnn/2120">Simple working example how to use packing for variable-length sequence inputs for rnn</a> 
I have followed the pytorch documentation and coded with batch First 
import torch
import torch.nn as nn
from torch.autograd import Variable

batch_size = 3
max_length = 3
hidden_size = 2
n_layers =1
num_input_features = 1
input_tensor = torch.zeros(batch_size,ma…
  </blockquote>
</aside>
<aside class="quote" data-post="1" data-topic="22242">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/7993a0/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-isnt-pad-packed-sequence-sufficient-why-do-we-need-to-pad-by-ourselves-when-we-supply-the-sequence-lengths/22242">Why isn't pad_packed_sequence sufficient ? Why do we need to pad by ourselves when we supply the sequence lengths?</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    Why do we need to pad the input for variable length sequences for lstm when there is a pack_padded_sequence function that essentially tells the lstm to ignore the padded portion? Why isn’t the pack_padded_sequence function with the sequence lengths sufficient for training via mini batches? Why do we need to pre-pad the input ?
  </blockquote>
</aside>

<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.quora.com/Why-do-sequences-need-to-get-sorted-for-packing">
  <header class="source">
      <img src="https://qsf.cf2.quoracdn.net/-4-images.favicon-new.ico-26-07ecf7cd341b6919.ico" class="site-icon" width="" height="">

      <a href="https://www.quora.com/Why-do-sequences-need-to-get-sorted-for-packing" target="_blank" rel="noopener nofollow ugc">Quora</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://qsf.cf2.quoracdn.net/-4-images.Illustration_fb_share_default_1280x720.png-26-7b8bc71e8107b6f5.png" class="thumbnail">

<h3><a href="https://www.quora.com/Why-do-sequences-need-to-get-sorted-for-packing" target="_blank" rel="noopener nofollow ugc">Why do sequences need to get sorted for packing?</a></h3>

  <p>Answer: I think it mostly has to do with minimization of the size of the Tensors.

In general - when we have the Products of the Matris computations/Tensor computations - you need a as compact Tensor as possible.

The simple reason for this - is to...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="quote quote-modified" data-post="1" data-topic="13268">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/josmi9966/48/2272_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-do-sequences-need-to-get-sorted-for-packing/13268">Why do sequences need to get sorted for packing?</a> 
  </div>
  <blockquote>
    In order to make an LSTM deal with variable length sequences in a batch, I have to use pack_padded_sequence first. But why do the sequences for a batch have to be sorted in decreasing order of length? In order to match each of the sequences with the correct training signal, I then have to unsort the output of the unpacked sequences again. I do not understand why this reordering is necessary, it makes everything a whole lot more complex in the code. 
What I also do not understand is why there is …
  </blockquote>
</aside>