Using cross_entropy's ignore_index, instead of pack_padded_sequence

antoniogois: <p>Suppose I’m using <code>cross_entropy</code> loss to do language modelling (to predict the next element in a sequence).</p>
<p>I have sequences with different lengths that I want to batch together, and the usual solution is to order them, pad with a special symbol (say 0), then use <code>pack_padded_sequence()</code>, feed them to an RNN and then <code>.pad_packed_sequence()</code>.</p>
<p>However, for the loss function <code>.cross_entropy()</code> to ignore the padding, I should also use <code>.cross_entropy(output_padded, ignore_index=0)</code>.</p>
<p>My question is – what happens if I do all of this, but without using <code>pack_padded_sequence()</code> and <code>.pad_packed_sequence()</code>? If <code>.cross_entropy(output_padded, ignore_index=0)</code> already ignores all 0’s, and assuming that 0’s only occur at the end of sequences, I don’t see how the outcome should differ.</p>

mariosasko: <p>They are 2 different concepts.</p>
<p>Sequences are packed to ignore padding and improve the efficiency and even accuracy (intuitively, padding adds noise) in return.</p>
<p>And the loss function is related to the final prediction. Here you no longer care about padding, you need padding just to pass the RNN step. <code>ignore_index=0</code> in CrossEntropyLoss means if true label is 0 then this particular loss is ignored (does not contribute to the already accumulated loss).</p>
<p>In the example below, the first instance (<code>logits[0]</code>) does not contribute to the final loss because its true label is 0 and we ignore such instances:</p>
<pre><code class="lang-python">torch.nn.CrossEntropyLoss(ignore_index=0)(torch.randn(2, 3), torch.tensor([0, 2]))
</code></pre>

antoniogois: <aside class="quote no-group" data-username="mariosasko" data-post="2" data-topic="99394">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mariosasko/48/27520_2.png" class="avatar"> mariosasko:</div>
<blockquote>
<p>Sequences are packed to ignore padding</p>
</blockquote>
</aside>
<p>But if all my zeros occur exactly in the end of each sentence, wouldn’t that imply that anything happening during the padding symbols (i.e. anything after the end of sentence) is ignored by the loss function?</p>
<p>In the particular setting I’m describing, I don’t see any difference between the two. One asks to ignore symbols=0, the other asks to ignore symbols after end-of-sentence (which are 0’s)</p>

antoniogois: <aside class="quote no-group" data-username="mariosasko" data-post="2" data-topic="99394">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mariosasko/48/27520_2.png" class="avatar"> mariosasko:</div>
<blockquote>
<p>And the loss function is related to the final prediction.</p>
</blockquote>
</aside>
<p>Maybe I should add that I have one final prediction for each element of the sequence (since we are trying to predict the next element, for each position in the sequence)</p>

mariosasko: <aside class="quote no-group" data-username="antoniogois" data-post="3" data-topic="99394">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/antoniogois/48/27180_2.png" class="avatar"> antoniogois:</div>
<blockquote>
<p>But if all my zeros occur exactly in the end of each sentence, wouldn’t that imply that anything happening during the padding symbols (i.e. anything after the end of sentence) is ignored by the loss function?</p>
</blockquote>
</aside>
<p>The computed loss on some position let’s call it <code>t</code> is ignored only if true label on that position <code>t</code> is equal to the specified <code>ignore_index</code>, so if you match true labels that are equal to <code>ignore_index</code> with padding positions then you can ignore those predictions (if you match them correctly somehow). There is no magic between these 2 concepts i.e. <code>nn.CrossEntropyLoss</code> and <code>ignore_index</code> are unrelated to padding.</p>
<p>It’s possible that I misunderstood you (there are all different kinds of language modeling). What’s your input/output?</p>

antoniogois: <p>ok, I’ve written a self-contained example to illustrate my point  (I couldn’t make it smaller than this)</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn import Embedding, LSTM
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
torch.manual_seed(0)

seqs = ['long_str', 'medium', 'tiny']  # already ordered by length
vocab = ['&lt;pad&gt;'] + sorted(set([char for seq in seqs for char in seq]))
inputs = [s[:len(s) - 1] for s in seqs]  # no prediction for final char
targets = [s[1:len(s)] for s in seqs]  # predict next char
vectorized_inps = [[vocab.index(tok) for tok in seq] for seq in inputs]
vectorized_tgts = [[vocab.index(tok) for tok in seq] for seq in targets]


class RnnLangModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = Embedding(len(vocab), 4)  # embedding_dim = 4
        self.lstm = LSTM(input_size=4, hidden_size=5, batch_first=True)  # input_dim = 4, hidden_dim = 5
        self.decoder = nn.Linear(5, len(vocab))  # output 1 logit for each element in vocab

    def forward(self, inp):
        embedded_padded_input = self.embed(inp)
        packed_input = pack_padded_sequence(embedded_padded_input, seq_lengths, batch_first=True)
        packed_output, (ht, ct) = self.lstm(packed_input)
        output, input_sizes = pad_packed_sequence(packed_output, batch_first=True)
        return self.decoder(output)

    def forward_no_packing(self, inp):
        embedded_padded_input = self.embed(inp)
        output, (ht, ct) = self.lstm(embedded_padded_input)
        return self.decoder(output)


rnn_lm = RnnLangModel()

# prepare input
seq_lengths = torch.LongTensor(list(map(len, vectorized_inps)))  # get the length of each seq
padded_input = torch.zeros((len(vectorized_inps), seq_lengths.max())).long()
for idx, (seq, seqlen) in enumerate(zip(vectorized_inps, seq_lengths)):
    padded_input[idx, :seqlen] = torch.LongTensor(seq)
# prepare output
padded_targets = torch.zeros((len(vectorized_tgts), seq_lengths.max())).long()
for idx, (seq, seqlen) in enumerate(zip(vectorized_tgts, seq_lengths)):
    padded_targets[idx, :seqlen] = torch.LongTensor(seq)

output = rnn_lm(padded_input)
output_no_packing = rnn_lm.forward_no_packing(padded_input)

ce_loss1 = F.cross_entropy(output.view(-1, output.shape[-1]),  # flatten output, except for logits dim
                          padded_targets.view(-1),
                          ignore_index=0)

ce_loss2 = F.cross_entropy(output_no_packing.view(-1, output.shape[-1]),
                           padded_targets.view(-1),
                           ignore_index=0)

ce_loss3_wrong = F.cross_entropy(output.view(-1, output.shape[-1]),
                          padded_targets.view(-1),
                          ignore_index=-100)

ce_loss1.backward()
# ce_loss2.backward()
# ce_loss3_wrong.backward()
</code></pre>
<p>Whenever the goal is to make 1 discrete prediction per sequence-element based only on previous context, we have a setup like the one here.</p>
<p>In these cases, sequences need to be padded, just like the targets.<br>
Therefore, when it’s time to compute cross-entropy, it requires using <code>ignore_index</code>, otherwise we end up with <code>ce_loss3_wrong</code>, even if we use <code>pack_padded_sequence</code>.</p>
<p>Since we have to use <code>ignore_index</code>, I was wondering what’s the need for using <code>pack_padded_sequence</code> at all. Apparently it’s none, since <code>ce_loss1</code> and <code>ce_loss2</code> provide the same loss and the same gradient updates to the model (if you inspect <code>output</code> there will be some differences but they are ignored by <code>F.cross_entropy</code>). There might still be some computation saved by using <code>pack_padded_sequence</code>, but I’m not even sure of that, since everything is done in parallel.</p>
<p>Now, I’m obviously not claiming that <code>pack_padded_sequence</code> is useless, it can be great for instance if we only need 1 final hidden state for each sequence. But in this setting (which is not so rare) it seems a bit redundant to use it.</p>
<p>Btw the code was loosely inspired by <a href="https://gist.github.com/HarshTrivedi/f4e7293e941b17d19058f6fb90ab0fec" rel="noopener nofollow ugc">this example</a>.</p>