Ignore padding area in loss computation

username: ptrch_c_m
cooked: <p>I am working on small texts doing Sequence Labelling. Specifically, I use a BERT model from the huggingface library (BertModel in particular), and I tokenize every text with the library’s tokenizer to feed the model. Since the texts are small, I have specified that the sequence length that the tokenizer produces is 256. My labels are binary (1 and 0) and every sequence element (BERT input token) is assigned a label.</p>
<p>For the loss computation I use Binary Cross Entropy (BCEWithLogitsLos) but the function considers also the padding tokens to compute the loss which also affects back propagation.</p>
<p>I want BCEWithLogitsLos to compute the loss only on the tokens of the text and not also on the padding tokens. Which is the best way to achieve that?</p>

username: ptrblck
cooked: <p>I think you could try to use the raw loss output (via <code>reduction='none'</code>), set the unwanted loss entries to zero, reduce the loss, and calculate the gradients via <code>loss.backward()</code>. Unsure, if there is a better way to mask the loss.</p>

username: ptrch_c_m
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your response.</p>
<p>You are right, this is a clean way to implement it.<br>
I tried it and even though the loss was different, the model metrics did not change.<br>
I suppose that when you said to reduce the loss, because <code>reduction='none'</code>, you meat to use <code>torch.mean()</code> after turning into zero the loss of the padded tokens.</p>
<p>I just mentioned that because the resulting tensor  <code>grad_fn</code> is <code>Mean</code>, although the backward computation will have all steps, including <code>BCEWithLogits</code>.</p>
<p>I also cannot think of a better way to implement this.</p>
<p>Regards.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ptrch_c_m" data-post="3" data-topic="95804">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e274bd/48.png" class="avatar"> ptrch_c_m:</div>
<blockquote>
<p>I just mentioned that because the resulting tensor <code>grad_fn</code> is <code>Mean</code> , although the backward computation will have all steps, including <code>BCEWithLogits</code> .</p>
</blockquote>
</aside>
<p>Yes, that’s what I had in mind. The backward call should still work as intended and internally the mean reduction would do the same. In your manual approach the <code>grad_fn</code> would point to the mean operation, which shouldn’t be a concern.</p>

username: ptrch_c_m
cooked: <p>That is also what I thought. Thanks again for your help.</p>
<p>Regards.</p>

username: izaskr
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, I have a follow-up question to that.<br>
Using <code>torch.mean()</code> implies that the elements that were zeroed out are also taken into account when calculating the average and thus affect the backpropagation. I’m wondering if it makes more sense to divide by the count of non-zeroed out elements.</p>
<pre><code class="lang-auto">pad = 2

tags = torch.tensor([0,1,1,0,1,2])

# for this example, let's pretend this is our loss tensor that we got from the unreduced BCEWithLogitsLoss
loss = torch.tensor([0.001, -0.3, 0.9, 0.7,0.6, 0.8])  

loss_mask = tags != pad
# loss_mask tensor([ True,  True,  True,  True,  True, False])

loss_masked = loss.where(loss_mask, torch.tensor(0.0))
# loss_masked tensor([ 0.0010, -0.3000,  0.9000,  0.7000,  0.6000,  0.0000])

loss_masked.mean()  # tensor(0.3168)

loss_masked.sum() / loss_mask.sum()  # tensor(0.3802)
</code></pre>

username: ptrblck
cooked: <p>Yes sounds like an interesting idea! In case you are using this approach and have a comparison to the <code>mean</code> approach it would be interesting to see your findings and if you’ve seen any advantage using one or the other during training.</p>

username: aleph_one
cooked: <p>FYI I just compared the two approaches on an NMT problem  <a class="mention" href="/u/izaskr">@izaskr</a> was saying:</p>
<pre><code class="lang-auto"># option A
loss_masked.mean()  # tensor(0.3168)

# option B
loss_masked.sum() / loss_mask.sum()  # tensor(0.3802)
</code></pre>
<p>Option B works much better.  Monotonically decreasing loss, better generalization.</p>

username: saxenarohit
cooked: <p>An alternative way of option B</p>
<pre><code class="lang-auto">loss_masked = torch.masked_select(loss, loss_mask)
loss_masked.mean()

</code></pre>

username: braunagn
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> - can this also be done by specifying <code>ignore_index==pad_token_id</code> in the cross entropy function, vs having to do the zeroing-out manually? <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.cross_entropy — PyTorch 2.1 documentation</a></p>

username: ptrblck
cooked: <p>Yes, this should work if you are using <code>nn.CrossEntropyLoss</code> with class indices as the target. Note that the author used <code>nn.BCEWithLogitsLoss</code>, which might need a manual masking as “soft” targets are expected.</p>