Batching with padded sequences and pack_padded_sequence

username: maria
cooked: <p>I’m new to PyTorch, and trying implement a language model with an LSTM. I’ve checked out two very good posts explaining padding and pack_padded-sequences:<br>
<a href="https://suzyahyah.github.io/pytorch/2019/07/01/DataLoader-Pad-Pack-Sequence.html" class="onebox" target="_blank" rel="nofollow noopener">https://suzyahyah.github.io/pytorch/2019/07/01/DataLoader-Pad-Pack-Sequence.html</a><br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="16" height="16">
      <a href="https://towardsdatascience.com/taming-lstms-variable-sized-mini-batches-and-why-pytorch-is-good-for-your-health-61d35642972e" target="_blank" title="12:29AM - 05 June 2018" rel="nofollow noopener">Medium – 5 Jun 18</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:129/62;"><img src="https://miro.medium.com/freeze/max/500/1*_odqFvts1GBGFgddcmcGvg.gif" class="thumbnail"></div>

<h3><a href="https://towardsdatascience.com/taming-lstms-variable-sized-mini-batches-and-why-pytorch-is-good-for-your-health-61d35642972e" target="_blank" rel="nofollow noopener">Taming LSTMs: Variable-sized mini-batches and why PyTorch is good for your...</a></h3>

<p>If you’ve used PyTorch you have likely experienced euphoria, increased energy and may have even felt like walking in the sun for a bit…</p>

  <p><span class="label1">Reading time: 6 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>However, I don’t see how I can pass the input length variable when batching, as I need it for the pack_padded_sequence.</p>
<p>Also, I would appreciate an explanation on the collate_fn in batching.</p>

username: G.M
cooked: <p>I’m not clear about what r u trying 2 do in Q1, can u explain more?</p>
<p>collate_fn is the function used to combine a bunch of samples taken from a Dataset to something that can be fed into a module. For instance, if your dataset returns a single tensor with one index, then an acceptable collate_fn could be torch.cat or something that combines these tensors into a batch(usually a bigger tensor).</p>

username: maria
cooked: <p>After padding, I will need to use something like the following (from the 2nd link):</p>
<pre><code class="lang-auto">        X = torch.nn.utils.rnn.pack_padded_sequence(x, **X_lengths**, batch_first=True)

        # now run through LSTM
        X, self.hidden = self.lstm(X, self.hidden)

        # undo the packing operation
        X, _ = torch.nn.utils.rnn.pad_packed_sequence(X, batch_first=True)
</code></pre>
<p>The X_length parameter is needed for the RNN to ignore the padded parts, and it’s passed to the forward function. I have no problem calculating it for the entire dataset, but how do I pass it if I choose to batch the data?<br>
I though of using the collate function and returning the length for each batch, but than how is it passed to the forward function of the module?</p>

username: G.M
cooked: <p>You r correct, it is usually done through the collate_fn. Just change the default collate_fn of DataLoader to your collate_fn and u r good to go. DataLoader first get the samples from the Dataset, pass it to the collate_fn and return it each iteration. So what your collate_fn returns is what you get from the DataLoader.</p>

username: hadaev8
cooked: <p>Check this example<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/NVIDIA/tacotron2/blob/master/data_utils.py#L80" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/NVIDIA/tacotron2/blob/master/data_utils.py#L80" target="_blank" rel="nofollow noopener">NVIDIA/tacotron2/blob/master/data_utils.py#L80</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="70" style="counter-reset: li-counter 69 ;">
<li>def __init__(self, n_frames_per_step):</li>
<li>    self.n_frames_per_step = n_frames_per_step</li>
<li>
</li>
<li>def __call__(self, batch):</li>
<li>    """Collate's training batch from normalized text and mel-spectrogram</li>
<li>    PARAMS</li>
<li>    ------</li>
<li>    batch: [text_normalized, mel_normalized]</li>
<li>    """</li>
<li>    # Right zero-pad all one-hot text sequences to max input length</li>
<li class="selected">    input_lengths, ids_sorted_decreasing = torch.sort(</li>
<li>        torch.LongTensor([len(x[0]) for x in batch]),</li>
<li>        dim=0, descending=True)</li>
<li>    max_input_len = input_lengths[0]</li>
<li>
</li>
<li>    text_padded = torch.LongTensor(len(batch), max_input_len)</li>
<li>    text_padded.zero_()</li>
<li>    for i in range(len(ids_sorted_decreasing)):</li>
<li>        text = batch[ids_sorted_decreasing[i]][0]</li>
<li>        text_padded[i, :text.size(0)] = text</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>