GRU and padded sequences | Tipps and Tricks

username: Unity05
cooked: <p>Hi,</p>
<p>according to my understanding of GRUs, extending a sequence with zeros (-&gt; sequence padding) should not make a huge difference in the final output, as long as the padded length is not too long.<br>
I had a few problems in my latest network, until I figured out, that the difference between the several sequences was too big.<br>
My first idea was to sort the sequences by their length, so that the total length difference in a padded batch would be as small as possible. Now I’m wondering whether there is an even better way to prevent the sequence padding to falsifie the networks output (by too much).<br>
Thank you.</p>
<p>Regards,<br>
Unity05</p>

username: googlebot
cooked: <p>It is not necessary to use outputs produced using padding zeros.</p>
<ol>
<li>
<p>if you’re using hidden state as RNN result, <a href="https://pytorch.org/docs/stable/nn.html#pack-padded-sequence" rel="nofollow noopener">packed sequences</a> support variable length sequences</p>
</li>
<li>
<p>if you’re using regular output (with time dimension), you can create a loss mask from sequence lengths, so that padding locations produce zero gradients.</p>
</li>
</ol>

username: vdw
cooked: <aside class="quote no-group" data-username="Unity05" data-post="1" data-topic="90729">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/unity05/48/20944_2.png" class="avatar"> Unity05:</div>
<blockquote>
<p>My first idea was to sort the sequences by their length, so that the total length difference in a padded batch would be as small as possible. Now I’m wondering whether there is an even better way to prevent the sequence padding to falsifie the networks output (by too much).</p>
</blockquote>
</aside>
<p>The <a href="https://torchtext.readthedocs.io/en/latest/data.html#bucketiterator" rel="noopener nofollow ugc"><code>BucketIterator</code></a> creates batches with sequences of the same or similar lengths. Or you can create your own Sampler to create batches with sequences of the same length; see <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284/13">this post as an example</a>.</p>
<p>The comments from <a class="mention" href="/u/googlebot">@googlebot</a> are of course valid, but it’s generally a good idea to avoid batches where the lengths of the sequences vary a great deal.</p>

username: Unity05
cooked: <p>Thank you for your replies!<br>
I guess Chris’ answer is the more general solution.<br>
However, in my case, I think masking should work better. Just for being sure: using the regular output, I should adjust my targets so that they are of size <code>(n_batches, padded_len)</code> by filling them with sequence_len times the class_index and pad them together. So assuming I use pad value 0 for my targets, I should then use <code>ignore_index=0</code> in my Loss?<br>
I’m just asking this as I’m a bit curious about the targets. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>Regards,<br>
Unity05</p>

username: googlebot
cooked: <p>I haven’t tried ignore_index, that method will probably work. More general way is to use reduction=‘none’, multiply padded loss cells by zero and reduce loss tensor using weighted mean (i.e. divide loss sum by number of good cells).</p>

username: Unity05
cooked: <p>I see your point. But according to the docs, these two methods should not make any difference, should they?</p>
<pre><code class="lang-auto">ignore_index (int, optional) – 
Specifies a target value that is ignored 
and does not contribute to the input gradient. 
When size_average is True, 
the loss is averaged over non-ignored targets.
</code></pre>
<p>Setting <code>ignore_index</code> to the padding value of the targets, their losses will be ignored. It also says the with <code>size_average=True</code>, the mean is based on the non - ignored losses.<br>
However, I’m curious how this behaves with <code>reduction</code> as <code>size_average</code> is going to be deprecated.</p>
<p>Regards,<br>
Unity05</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="Unity05" data-post="6" data-topic="90729">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/unity05/48/20944_2.png" class="avatar"> Unity05:</div>
<blockquote>
<p>I see your point. But according to the docs, these two methods should not make any difference, should they?</p>
</blockquote>
</aside>
<p>yeah, but there is no ignore_index for regression losses…</p>

username: Unity05
cooked: <p>That’s true. Therefore, when working with regression losses, your solution is the way to choose. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>