Calculating loss on sequences with variable lengths

username: shavitamit
cooked: <p>I’m doing a simple seq2seq encoder-decoder model on batched sequences with varied lengths, and I’ve got it working with the <code>pack_padded_sequence</code> and <code>pad_packed_sequence</code> for the encoder.</p>
<p>Now, after decoding a batch of varied-length sequences, I’d like to accumulate loss <strong>only</strong> on words in my original sequence (i.e., not on <code>&lt;PAD&gt;s</code>)</p>
<p><strong>Originally, I was accumulating loss on the entire batch like so:</strong></p>
<pre><code>loss_function = nn.NLLLoss()
loss = 0
for word in range(max_seq_len_in_batch - 1):
    loss += loss_function(output[:,word,:], y_data[:,word+1])
</code></pre>
<p>However, that doesn’t take into account variable length decoded sequences.  I don’t want to accumulate loss on <code>&lt;PAD&gt;</code> elements.  I then changed the loop to:</p>
<pre><code># -- for each sequence in the batch
for idx, seq_len in enumerate(ylen):
    
    # -- for each word in the sequence
    for word in range(seq_len-1):       
        loss += loss_function(output[idx,word,:].unsqueeze(0), y_data[idx,word+1])
</code></pre>
<p>Granted, that gives a different <code>loss</code> value than the previous way of calculating it on the entire batch.  However, it’s drastically different, by orders of magnitude.  I’m wondering if that will affect the backpropagation…</p>
<p><strong>Question 1</strong>: Am I doing the above correctly? Should I normalize somehow?<br>
<strong>Question 2</strong>: What is the correct way to accumulate loss on sequences with variable lengths in batches?</p>
<p>Thanks.</p>

username: colesbury
cooked: <p>They’re different because by default NLLLoss averages over the number of observations. See:</p>
<p><a href="http://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss" class="onebox" target="_blank">http://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss</a></p>
<p>Set size_average to <code>False</code> and divide the loss by the number of non-padding tokens. That should give you approximately the same value.</p>
<p>It will affect back-propagation in the same way that scaling your learning rate affects it: scaling the loss by X scales the gradients by the same factor X.</p>

username: shavitamit
cooked: <p>Thanks, that helps. One more thing I’ve noticed is that by adding that<br>
second loop, my time per epoch is slowed down by almost four-fold.</p>
<p>Are there ways to make that calculation more efficient?</p>

username: colesbury
cooked: <p>NLLLoss has an <code>ignore_index</code>. Use the batched version, but set <code>ignore_index</code> to your padding value.</p>

username: shavitamit
cooked: <p>Thanks - that solved it. Perfect.</p>

username: shavitamit
cooked: <p>By the way - is there a way to avoid the outer-most loop? (as shown below):</p>
<pre><code>loss_function = nn.NLLLoss(ignore_index=out_word2idx['&lt;PAD&gt;']).cuda()
loss = 0
for word in range(max_seq_len-1):
    loss += loss_function(output[:,word,:], y_data[:,word+1])    
</code></pre>
<p>I’m wondering if it’s possible to avoid the loop over individual words in the sequence altogether.</p>

username: colesbury
cooked: <p>Shouldn’t this work?</p>
<pre><code class="lang-auto">loss_function = nn.NLLLoss(ignore_index=out_word2idx['&lt;PAD&gt;']).cuda()
loss = loss_function(output, y_data)
</code></pre>
<p>FWIW, I find it generally cleaner to use the “functional” version, but it’s up to you:</p>
<pre><code class="lang-auto">import torch.nn.functional as F
loss = F.nll_loss(output, y_data, ignore_index=out_word2idx['&lt;PAD&gt;'])
</code></pre>

username: shavitamit
cooked: <p>I’ll give the “functional” version a try.  your first suggestion (<code>loss_function(output,y_data)</code>) does not work because I guess it expects either 2 or 4 dimensions.</p>
<p>One approach to doing it all in one line is actually like so:</p>
<pre><code>(torch.gather(-output[:,:-1,:],2,y_data[:,1:].unsqueeze(2)).squeeze().data * mask.cuda()).sum()
</code></pre>
<p>Where <code>mask</code> corresponds to a <code>[batch_sz, max_seq_len-1]</code> tensor of 1s and 0s corresponding to word or <code>&lt;PAD&gt;</code>, respectively.  That’s a pretty ugly solution in my opinion, and I haven’t even tried to check whether it’s faster or slower.  Plus, it requires the creation of a new <code>mask</code> tensor that needs to be copied to the GPU for each batch evaluation.</p>
<p>I’ll give your “functional” solution a try and hopefully that solves it.</p>

username: lyan62
cooked: <p>Hi did you work out a solution? I’m having a same problem</p>

username: shavitamit
cooked: <p>Which exact question are you referring to?  Not accumulating loss on <code>&lt;PAD&gt;</code> elements? If yes, I did solve that using the <code>ignore_index</code> argument (see above).</p>

username: aspiringcoder
cooked: <p>Hi <a class="mention" href="/u/shavitamit">@shavitamit</a>, I’m facing the same issue.</p>
<p>Target tensor of <code>BxS</code> where S is variable length padded with <code>&lt;PAD&gt;</code> token.</p>
<p>Output of my model is <code>BxV</code> where V is a probability distribution over the vocabulary.</p>
<p>I’ve currently ended up doing exactly what you did originally:</p>
<pre><code class="lang-python">crit = NLLoss(ignore_index='&lt;PAD&gt;')
for token in range(seqlen):
   pred,hidden = model(words[:,token],hidden)
   loss += crit(pred,target[:,token])
</code></pre>
<p>I’d love to figure out a smarter way to do it, and I do have a mask, flattening all my tokens without the padding token, but then the dimensions don’t add up, as the loss function expects a two dimensional input.</p>