How to use pack_padded_sequence correctly? How to compute the loss?

vdw: <p>I’m using a very simple RNN-based binary classifier for short text documents. As far as I cant tell, it works reasonable fine. The loss goess down nicely and the accuracy goes up over 80% (it plateaus after 30-40 epochs, I’m doing 100). The <code>forward</code> method of the classifier looks like this – the input batch X is sorted w.r.t. the their length but I don’t utilize it here:</p>
<pre><code>def forward(self, X_sorted, X_length_sorted, method='last_step'):
    X = self.word_embeddings(X_sorted)
    X = torch.transpose(X, 0, 1)
    X, self.hidden = self.gru(X, self.hidden)
    X = X[-1]
    # A series of fully connected layers
    for l in self.linears: 
        X = l(X)
    return F.log_softmax(X, dim=1)
</code></pre>
<p>Naturally, the length of sequences can vary between a minimum length of 5 and maximum length of 5. Now I wanted to see how the packing and padding of the sequences works. I therefore modified the <code>forward</code> method as follows:</p>
<pre><code>def forward(self, X_sorted, X_length_sorted, method='last_step'):
    X = self.word_embeddings(X_sorted)
    X = torch.transpose(X, 0, 1) 
    X = nn.utils.rnn.pack_padded_sequence(X, X_length_sorted)
    X, self.hidden = self.gru(X, self.hidden)
    X, output_lengths = nn.utils.rnn.pad_packed_sequence(X)
    X = X[-1]
    # A series of fully connected layers
    for l in self.linears:
        X = l(X)
    return F.log_softmax(X, dim=1)
</code></pre>
<p>The network still still trains, but I’ve noticed some differences</p>
<ul>
<li>Each epoch takes about 10-15% longer to process</li>
<li>The loss goes down much slower (using the same learning rate)</li>
<li>The accuracy goes up to only about 70% (it plateaus after 30-40 epochs, I’m doing 100)</li>
</ul>
<p>I also found to change <code>nn.NLLLoss()</code> to <code>nn.NLLLoss(ignore_index=0)</code> with <code>0</code> being the padding index. Again, it trains, but the loss goes down almost crazily fast (even with a much smaller learning rate) and the accuracy won’t change at all. I still somehow feel that the calculation of the loss is an issue.</p>
<p>In short, it kind of works in the sense that the networks train, but I fail to properly interpret the results Am I’m missing something here or are the expected results?</p>

Kushaj: <p>If you are concerned about model underfitting, you should try to overfit your model on a small mini-batch and see it the accuracy goes to 100%. If the model is not able to overfit then, you are underfitting.</p>

vdw: <p>Thanks! Using your idea I was able to drill down to the problem. Using a very small dataset, I could immediately overfit (training accuracy=100%) the model if I don’t use packing – didn’t happen when I used packing, initially. I finally got it to work with packing when I used a batch size of 1.</p>
<p>I’m pretty sure now, that I cannot simply used packing and <code>X=X[-1]</code> to get the last output. When batch size = 1 the GRU output dimension is (seq_len, batch_size, dim) where seq_len is only the length of the sequence without padding. If I have larger batches seq_len is length of the longest sequence in the batch. So when I do X=X[-1] I get meaningless output for all shorter sequences that have padding. I could confirm this by making sure that all my sequences in my mini dataset have no padding. Then I could overfit my model even with packing.</p>
<p>My current solution is there not last output but the final hidden state of the RNN. For this, used the approach outlined <a href="https://stackoverflow.com/questions/50856936/taking-the-last-state-from-bilstm-bigru-in-pytorch" rel="nofollow noopener">here</a>. Not 100% sure if this is the (most) correct way, but now I can train my model on the original dataset with packing and get the expected test accuracy of 80%.</p>

vdw: <p>After consulting the PyTorch Docs a bit longer and seeing some other code examples, I post below my current <code>forward</code> function. Maybe it’s useful for some people; I actually haven’t found that many examples for this.</p>
<pre><code>def forward(self, X_sorted, X_length_sorted, method='last_step'):
    X = self.word_embeddings(X_sorted)
    X = torch.transpose(X, 0, 1)
    X = nn.utils.rnn.pack_padded_sequence(X, X_length_sorted)
    X, self.hidden = self.gru(X, self.hidden)
    X, output_lengths = nn.utils.rnn.pad_packed_sequence(X)
    final_state = self.hidden.view(self.num_layers, self.directions_count, X_sorted.shape[0], self.rnn_hidden_dim)[-1]
    if self.directions_count == 1:
        X = final_state.squeeze()
    elif self.directions_count == 2:
        h_1, h_2 = final_state[0], final_state[1]  # forward &amp; backward pass
        #X = h_1 + h_2                # Add both states
        X = torch.cat((h_1, h_2), 1)  # Concatenate both states
    # A series of fully connected layers
    for l in self.linears:
        X = l(X)
    return F.log_softmax(X, dim=1)
</code></pre>
<p>Of course, the size of the first linear layer depends whether I sum or concatenate the hidden states in case of a bidirectional RNN. Using my simple dataset at the moment, both approaches work equally well, but I don’t know if one approach is generally preferable.</p>
<p>To recap the original problem: When using <code>PackedSequence</code>, one cannot simple use the last output of the RNN (in my code <code>X=X[-1]</code>), since the dimension of X after <code>pad_packed_sequence</code> is the size of the longest sequence in the batch. For shorter ones, the RNN does go that far.</p>

marrrcin: <p><a class="mention" href="/u/vdw">@vdw</a> thanks for sharing your piece of code it helped me a lot!</p>

marco_zaror: <p>Hi Chris <a class="mention" href="/u/vdw">@vdw</a>,</p>
<p>I’m having issues related to yours with packed sentences and I was wondering if you can help me. I’m  working on a very simple rnn model and I’ve got variable-length sentences for the input. On every example that I have seen in the past, they use rnn, gru or lstm, however I’m defining my own model so I don’t know how to use the packed sentence. Below is the relevant part of my code:</p>
<pre><code class="lang-auto">n_hidden = 200
   
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size
        self.x2h = nn.Linear(input_size, hidden_size)
        self.h2h = nn.Linear(hidden_size, hidden_size)
        self.h2o = nn.Linear(hidden_size, output_size)
        self.softmax = nn.Softmax(dim=1)
        
    def forward(self, input, hidden_):

        hidden1 = self.x2h(input)
        hidden2 = self.h2h(hidden_)
        hidden = hidden1 + hidden2
        output = self.h2o(hidden)
        output = self.softmax(output)
        
        return output, hidden
    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

rnn1 = RNN(n_vocab, n_hidden, n_vocab)

xpacked = torch.nn.utils.rnn.pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False)

h = rnn1.initHidden()
output, hidden = rnn1(xpacked, h)
</code></pre>
<p>This code throws an error and I suspect that is because I’m trying to pass all the words at once. The problem is I don’t know how to access to each word in the packed and still exploits the advantages of the function.<br>
My train function, that was working before the packing, is as follows:</p>
<pre><code class="lang-auto">def train(text_x_tensor1,label1):#, text_x_tensor2, label2):
    text_x_tensor1, label1 = text_x_tensor1.to(device), label1.to(device)
    rnn1.train()
    hidden_1 = rnn1.initHidden()
    hidden_1 = hidden_1.to(device)
    text_x_tensor1 = text_x_tensor1.permute(1,0,2)
    for i in range(len(text_x_tensor1)): #For each word
        output_1, hidden_1 = rnn1(text_x_tensor1[i], hidden_1) 
    loss1 = criterion(output_1,label1)
    optimizer1.zero_grad()
    loss1.backward()
    torch.nn.utils.clip_grad_norm_(rnn1.parameters(),1)
    optimizer1.step()
    return output_1, loss1,hidden_1
</code></pre>
<p>Hoping you can help me,<br>
Marco</p>

vdw: <p><a class="mention" href="/u/marco_zaror">@marco_zaror</a> I don’t think I can really help here since it require insights into the internals. I’m just the occasional users for my research work.</p>
<p>The problem is that you define your own RNN but are using a PyTorch data structure <code>PackedSequence</code>. This is arguably designed to work well with <code>nn.LSTM</code> and <code>nn.GRU</code>. Sure, in principle, it should be able to use it in a custom fashion, but I have no idea how. The questions is also if it’s worth the effort and re-invent the wheel – I understand, of course, that you’re (partly) doing this for education/understanding.</p>
<p>To be honest, I would ignore that issue. Just use the <code>BucketIterator</code> that creates batches where all sequences within a batch have the same or at least very similar length. Even if there’s padding, it’s minimal, so it arguably won’t have any negative effects. Or enforce batch with sequences of equal length; see <a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">this thread</a>.</p>

marco_zaror: <p>Thank you very much for your answer Chris <a class="mention" href="/u/vdw">@vdw</a>. Your idea is actually easier so thank you for that!<br>
If you let me, can I ask you one more question? I’ve been reading a lot about BPTT and now I’m confused with something else. As it can be seen on my train function, I compute the forward pass of my model n times (where n is the number of words in the sentence). After that, I only compute the loss (and back propagate) for the last word of the sentence, is that ok in your opinion?<br>
I’ve seen math explanations, and they imply that I should compute the loss for every word in the sentence and then average that loss (In other words, take the word 1 and predict word 2, then take words 1 and 2 and predict word number 3 and so on), but I’m not sure about that aproach.</p>
<p>Can I have your thoughts about it? I’m really sorry is the question is too basic…</p>

vdw: <p>Disclaimer: The following are just my thoughts. I’m neither in expert in deep learning nor PyTorch. It’s just one area related to my research work.</p>
<blockquote>
<p>I only compute the loss (and back propagate) for the last word of the sentence</p>
</blockquote>
<p>I’m rather sure that you backpropagate through the whole sequence. The <code>for</code> loop in your <code>train()</code> method should build the backgraph, so when you call <code>backward()</code> it should consider the whole graph.</p>
<blockquote>
<p>I’ve seen math explanations, […]</p>
</blockquote>
<p>Not sure what you mean by this last paragraph. You use the RNN for classification, so there’s no notion of “take the word 1 and predict word 2, then take words 1 and 2 and predict word number 3 and so on”. Since sounds more like sentence generation with some kind of decoder architecture.</p>

marco_zaror: <p>That’s exactly what I was thinking. I really appreciate your thoughts Chris <a class="mention" href="/u/vdw">@vdw</a>, thank you very much!</p>

MRLoghmani: <p>Thank you for your post!<br>
So far, I have failed to find a full example of training a recurrent net using pack_padded_sequence. I was wondering if there is anything we need to do in the backward step or if it remains the same as what it would be without packing.</p>