Using batches for Seq2Seq models

username: vdw
cooked: <p>Since I got a couple of questions in this <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284">previous thread</a>, which aims to order sequence data into batches where all input sequences in a batch have the same length. This avoids the need of padding and optional packing.</p>
<p>The original solution work only for sequence classification, sequence tagging, autoencoder models since the ordering only considered the input sequences. For Seq2Seq models such as machine translation, where the output sequences also differ in lengths, it didn’t work.</p>
<p>I’ve now made the small changes to support this as well – here is the code for the <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/utils/data/text/dataset.py" rel="noopener nofollow ugc">Sampler</a> and my own <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/utils/data/text/vectorizer.py" rel="noopener nofollow ugc">Vectorizer</a> I use, as well as a <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/notebooks/bucketbatchsampler-example.ipynb" rel="noopener nofollow ugc">Jupyter Notebook</a> showing the usage. Now, each batch contains sequence pairs where all input sequences are the same length and all target sequences have the same length. For example a batch might contain 32 pairs with all input sequences having length 12 and all target sequences having length 15.</p>
<p>To see how the combination of input and target lengths are distributed, I’ve made a test with a real-world <a href="http://www.isi.edu/natural-language/download/hansard/hansard.36.r2001-1a.house.debates.training.tar" rel="noopener nofollow ugc">dataset for machine translation</a>. I’ve set the batch size to 32. The figure below shows the distribution of the sizes of all batches. As one can see, the vast majority of batches is indeed full (i.e., 32 sequence pairs). This shouldn’t really be surprising since:</p>
<ul>
<li>Batch sizes (e.g., 32 or 64) are is essentially nothing given large datasets of millions of sequences pairs or more. Thus, the chance that enough pairs share the same input and target lengths is high.</li>
<li>The combination of input and target lengths is typically not independent. For example, an input of length 5 generally does not have a target length of 20 but in a similar range.</li>
</ul>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/1d57c6913cf581dc7d70b8164dece045b1f85679.png" alt="bucketbatchsample-batch-size-distribution" data-base62-sha1="4bzU7fVG3V3ojNGZBgf0iHYKzqN" width="432" height="288"></p>
<p>In short, I think it’s a convenient solution for working with batches in case of Seq2Seq models, again, with no need for padding and packing – and way faster that training with a batch size of 1. Maybe it’s useful for some of you.</p>

username: ptrblck
cooked: <p>Thanks for sharing this, Chris! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: TrentBrick
cooked: <p>Nice editions! <a class="mention" href="/u/ptrblck">@ptrblck</a> this should be added to Pytorch as a DataLoader object!</p>

username: ptrblck
cooked: <p>This might be a good idea and worth discussing.<br>
<a class="mention" href="/u/vdw">@vdw</a> would you mind creating an issue in the GitHub repo and linking your post here?</p>

username: vdw
cooked: <p>As suggested, I’ve submitted a feature requested. However, I feel the latest version of the <code>BucketIterator</code> of <code>torchtext</code> might be already sufficient. At least, I saw some examples online where it was used for machine translation tasks. While it “only” minimizes the padding, one can argue that this hardly effects the accuracy of the model anymore. I haven’t checked the inner workings of the <code>BucketIterator</code>, though.</p>
<p><strong>EDIT:</strong> <a href="https://github.com/pytorch/pytorch/issues/25743" rel="nofollow noopener">Link to the feature request</a> in case someone wants to comment on.</p>