Padding sequence in LSTM

username: chinmay5
cooked: <p>I have a few doubts regarding padding sequences in a LSTM/GRU:-</p>
<ul>
<li>
<p>If the input data is padded with zeros and suppose 0 is a valid index in my Vocabulary, does it hamper the training</p>
</li>
<li>
<p>After doing a pack_padded_sequence , does Pytorch take care of ensuring that the padded sequences are ignored during a backprop</p>
</li>
<li>
<p>Is it fine to compute loss on the entire padded sequence</p>
</li>
<li>
<p>While evaluating, I use value of the final hidden state <code>h_n</code> as the one for my prediction tasks. Is this value already corrupted by my input sequence padding?</p>
</li>
</ul>

username: vdw
cooked: <ul>
<li>If you use 0 for padding, then it shouldn’t be also used for any word in your vocabulary. Otherwise the model cannot distinguish between padding and the respective word.</li>
<li>
<code>pack_padded_sequence</code> should take care of correctly backpropagating. That’s the overall purpose</li>
</ul>
<p>Regarding the last 3 points in general: You might want to consider creating batches where all sequences within one batch have the same length. No need for padding or packing; <a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">see this older post</a>. It’s simply convenient, and I use it all the time.</p>

username: chinmay5
cooked: <p><a class="mention" href="/u/vdw">@vdw</a> thank you so much for the reply. I am using glove pretrained embeddings. Do you think it would be a better idea to start using ‘unk’ token for padding? For the time being, I started adding ‘eos’ token at the end of each sentence. My idea is, network would learn to ignore all padded tokens beyond this ‘unk’ token. Since I am still new to using lstm, your help would be highly appreciated</p>

username: zhangguanheng66
cooked: <p>I think <code>unk</code> and <code>pad</code> have different purpose. For example, you might have some unknown word in your test/evaluate datasets while the vocabulary is built on training dataset. In that case, you want to use <code>unk</code> for the unknown words. For padding, it’s obviously for the scenario when the sequences have different lengths.<br>
Anyway, it would be an interesting experiment to <code>using unk token for padding</code>.</p>

username: vdw
cooked: <p>As <a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a> rightly said, <code>&lt;unk&gt;</code>  and <code>&lt;pad&gt;</code> serve different purposes.<code>unk</code> represents tokens that are not in your vocabulary, while <code>&lt;pad&gt;</code> is some kind of non-token to create batches with sequences of equal length. Some very simple consquences:</p>
<ul>
<li>
<code>&lt;pad&gt;</code> can only be at the end or start of sequence (depending which “side” you pad), while <code>&lt;unk&gt;</code> can be anywhere in the non-padded part of the sequence</li>
<li>If, say, you pad at the end, there can never be an <code>unk</code> (or any other token) after the first <code>&lt;pad&gt;</code>
</li>
<li>Since <code>pad</code> are kind of filler tokens they are potentially not great to have; it’s not obvious (to me) how <code>&lt;pad&gt;</code> effects the learning. You generally cannot avoid <code>&lt;unk&gt;</code>.</li>
</ul>
<p>In short, there are arguably differences between <code>&lt;unk&gt;</code> and <code>&lt;pad&gt;</code>. Sure, you can use the same index value to represent both tokens – and maybe it doesn’t cause any problems – but I wouldn’t rob the network of the knowledge that these two token serve different purposes.</p>
<p>Personally, I never use padding (and packing), but <a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">organize my batches in such a way that all sequences have the same length</a>. This has a better performance, and I don’t have to worry whether <code>&lt;pad&gt;</code> effects the learning. You also might want to look at PyTorch’s <code>BucketIterator</code> which minimizes padding out of the box.</p>

username: chethanjjj
cooked: <p>For padding, in your experience, do you recommend padding at the start or end? I’ve seen discussions for both.</p>

username: Brando_Miranda
cooked: <p>Hi Chris <a class="mention" href="/u/vdw">@vdw</a>,</p>
<p>I was wondering, how have your practices for padding changed since the rise of the transformer?</p>
<p>It seems to me that the best thing to do in that case it to pad with zero vector (equal to the size of the embed dim). That way with the self-attention, when we compare a real token and a padd token (which is just zero) the output attention results in being zero. Is this not what is done?</p>

username: Brando_Miranda
cooked: <p>Nvm, you take care of my intended padding with a -inf because of how the softmax works in MHA, see: <a href="https://discuss.pytorch.org/t/how-to-add-padding-mask-to-nn-transformerencoder-module/63390" class="inline-onebox">How to add padding mask to nn.TransformerEncoder module?</a></p>