Loss for output sequence of variable length

kfshr: <p>I’m working on a project where I want to train a model to predict a sequence of tokens. For a given input sequence, the size of the output sequence can be between 1 and 4 tokens. As the size is variable, I use an “empty” token, here index 0, as padding.</p>
<p>The following lists show four sequences of length 3, 1, 2, 4 padded with 0s so that all sequences have final length of four.</p>
<p><code>[[5, 1, 4, 0,], [2, 0, 0, 0], [4, 3, 0, 0], [2, 1, 3, 8]]</code></p>
<p>I’m using the <code>CrossEntropyLoss</code> where I set <code>ignore_index=0</code>. I’m wondering if this is correct, as there are no error signals coming from the padded regions.</p>
<p>I would like to know how I use the <code>CrossEntropyLoss</code> correctly for sequences of variable length that are padded.</p>

KFrank: <p>Hi kfshr!</p>
<aside class="quote no-group" data-username="kfshr" data-post="1" data-topic="183601" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/45deac/48.png" class="avatar"> kfshr:</div>
<blockquote>
<p>I use an “empty” token, here index 0, as padding.</p>
</blockquote>
</aside>
<p>Use <code>-100</code> as your “empty” token.  This is <code>CrossEntropyLoss's</code> default<br>
for its <code>ignore_index</code> constructor argument.  (If you prefer a different value<br>
for some reason, you can use <code>ignore_index</code> explicitly.)</p>
<p>Items in your <code>target</code> (your ground-truth labels) that have a value of <code>-100</code><br>
will be ignored in the loss computation.</p>
<p>Best.</p>
<p>K. Frank</p>

kfshr: <p>Why is <code>-100</code> the default  “empty” token? Why not <code>-1</code> or some other value? Is there some reason? (I’m just curious).</p>
<p>I’m wondering if your answer is a “yes” to my question? I’m aware that I can use <code>ignore_index</code> to ignore items in the loss computation. I’m just wondering if this is the right approach for problems with variable output length? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi kfshr!</p>
<p>Sorry, I misread your original post.</p>
<aside class="quote no-group" data-username="kfshr" data-post="3" data-topic="183601" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/45deac/48.png" class="avatar"> kfshr:</div>
<blockquote>
<p>Why is <code>-100</code> the default  “empty” token? Why not <code>-1</code> or some other value? Is there some reason? (I’m just curious).</p>
</blockquote>
</aside>
<p>Just speculation:  <code>-100</code> looks sufficiently out of place as an integer<br>
categorical class label that one would likely suspect that it is some<br>
sort of sentinel value.  You wouldn’t want to use, say, <code>17</code> or <code>99</code> or even<br>
<code>999</code> as a sentinel value as these could be legitimate class labels for<br>
plausible use cases.  <code>-1</code> would not typically look like a class label (and<br>
would not be valid for pytorch’s <code>CrossEntropyLoss</code>), but you could<br>
imagine a class of bugs that might give you <code>-1</code> by accident, and if <code>-1</code><br>
were the default for <code>ignore_index</code>, such an error would not be caught.</p>
<p>(Your choice of <code>ignore_index = 0</code> is acceptable and logically valid.<br>
I would, however, recommend not using it, both for robustness (see<br>
above) and stylistic (e.g., stick with the default) reasons.)</p>
<aside class="quote no-group">
<blockquote>
<p>I’m wondering if your answer is a “yes” to my question? I’m aware that I can use <code>ignore_index</code> to ignore items in the loss computation. I’m just wondering if this is the right approach for problems with variable output length?</p>
</blockquote>
</aside>
<p>I think that the answer to your question could well be “no” (but it depends<br>
on your specific use case).</p>
<p>In your original post you say “For a given input sequence, the size of the<br>
output sequence can be between 1 and 4 tokens.”</p>
<p>This makes it sound like “for a given input sequence” your network needs<br>
to “learn” what length its predicted output sequence should be.</p>
<p>If this is your use case, I would use <code>0</code> as a <em>valid</em> class label (that is, not<br>
the <code>ignore_index</code> value).</p>
<p>Let’s say that your ground-truth label for the output sequence is<br>
<code>[2, 0, 0, 0]</code>, meaning that the correct sequence is of length one  and<br>
the first and only token is <code>2</code>.  A prediction of <code>[2, 0, 0, 0]</code> would be spot<br>
on, while a prediction of <code>[2, 1, 3, 8]</code> would be incorrect because it<br>
predicts a sequence length of four rather than one.</p>
<p>If you use <code>0</code> as a valid class label that means “placeholder for short<br>
sequence,” such a prediction will be penalized (which you most likely<br>
want).  If, instead, <code>0</code> is used as the <code>ignore_index</code> value, the loss values<br>
for the prediction <code>[2, 0, 0, 0]</code> and the prediction <code>[2, 1, 3, 8]</code> will be<br>
the same (because the last three predicted sequence tokens are ignored),<br>
which is most likely not what you want, because the incorrect prediction for<br>
the sequence length made by <code>[2, 1, 3, 8]</code> will not be penalized.</p>
<p>If this doesn’t make sense for your use case, could you give a little more<br>
detail about it, in particular whether and how your loss function should<br>
distinguish between correct and incorrect sequence-length predictions?</p>
<p>Best.</p>
<p>K. Frank</p>

kfshr: <p>This makes perfect sense. Using <code>ignore_index</code> doesn’t make sense after reading your thoughts on this topic. Thank you!</p>