Pytorch Chatbot loss function with ignore_index instead of target's padding mask

kedar: <p>Hello. I was going through Pytorch official tutorial regarding the implementation of seq2eq Chatbot with Attention. To gain better understanding I used different methods to achieve similar results. One of as I thought improvements was the implementation of simple nn.NLLLoss function with attribute ignore_index = padding_token. Previous solution that used binary mask tensor describing the padding of the target tensor seemed to be too complicated, I wanted a simpler solution, but as it tuned out adding few lines of code made everything worse.</p>
<p><code>criterion = nn.NLLLoss(ignore_index=PAD_token, reduction='mean')</code></p>
<p>In the definition of train function I only made small change in the loop when we forward propagate through decoder one step at a time:</p>
<pre><code>if use_teacher_forcing:
    for t in range(max_target_len):
        decoder_output, decoder_hidden = decoder(
            decoder_input, decoder_hidden, encoder_outputs
        )
        # Teacher forcing: next input is current target
        decoder_input = target_variable[t].view(1, -1)

        # Calculate and accumulate loss
      
        loss = criterion(decoder_output, target_variable[t])
        total_loss += loss
        print_losses.append(loss.item())


# Perform backpropatation
total_loss.backward()

# Clip gradients: gradients are modified in place
_ = nn.utils.clip_grad_norm_(encoder.parameters(), clip)
_ = nn.utils.clip_grad_norm_(decoder.parameters(), clip)

# Adjust model weights
encoder_optimizer.step()
decoder_optimizer.step()

return sum(print_losses)/max_target_len
</code></pre>
<p>Output:<br>
Iteration: 1; Percent complete: 0.0%; Average loss: 8.5246<br>
Iteration: 2; Percent complete: 0.1%; Average loss: 8.5403<br>
Iteration: 3; Percent complete: 0.1%; Average loss: 8.5663<br>
Iteration: 4; Percent complete: 0.1%; Average loss: 8.5691<br>
Iteration: 5; Percent complete: 0.1%; Average loss: 8.4637<br>
Iteration: 6; Percent complete: 0.1%; Average loss: 8.5378<br>
Iteration: 7; Percent complete: 0.2%; Average loss: 8.5575<br>
Iteration: 8; Percent complete: 0.2%; Average loss: 8.5145<br>
Iteration: 9; Percent complete: 0.2%; Average loss: 8.5717<br>
Iteration: 10; Percent complete: 0.2%; Average loss: 8.5122<br>
Iteration: 11; Percent complete: 0.3%; Average loss: 8.5587<br>
Iteration: 12; Percent complete: 0.3%; Average loss: 8.5491<br>
Iteration: 13; Percent complete: 0.3%; Average loss: 8.5933<br>
Iteration: 14; Percent complete: 0.4%; Average loss: 8.5231<br>
Iteration: 15; Percent complete: 0.4%; Average loss: 8.5239<br>
… an so on</p>
<p>Total loss increase or fluctuate ate the same value, but it doesn’t want to decrease. Did anybody try to use any kind loss function with ignore_index in seq2seq model with variate length sequences and has some positive results? nn.CrossEntropyLoss behaves similarly.</p>
<p>I would like to emphasize that tensors passed to loss function have the following dimensions:<br>
loss = criterion(decoder_output, target_variable[t])<br>
decoder_output: (batch_size, vocab_size)<br>
target_variable[t]: (batch_size)<br>
Maybe there is a problem? Are they a proper dimensions, or should I reshape these tensors? In original code they calculate negative log manually passing to it “concatenated” tensors with final shape: (batch_size):<br>
crossEntropy = -torch.log(torch.gather(inp, 1, target.view(-1, 1)).squeeze(1))</p>
<p>I tried to change learning rate, but it didn’t help at all.</p>
<p>One additional question. In my version I used:<br>
packed = nn.utils.rnn.pack_padded_sequence(embedded, input_lengths, batch_first=True, <strong>enforce_sorted=False</strong>), thus I didn’t sort batch in descending order according to sequence lengths. I noticed that my code process slower than original, could be that cause by using unsorted batch, or it should work well anyway?</p>
<p>Any suggestion would be very appreciated.</p>
<p>Best regards.</p>

zhangguanheng66: <p>I suggest you to open an issue on <a href="https://github.com/pytorch/tutorials" rel="nofollow noopener">pytorch/tutorials</a> and pin zhangguanheng66 and SethHWeidman.</p>

Abhilash_Srivastava: <p>Do you need to use <code>optimizer.zero_grad()</code> before <code>loss.backward()</code> ?</p>

kedar: <p>I don’t have to use <code>optimizer.zero_grad()</code> before <code>loss.backward()</code>, I can also use it after <code>loss.backward()</code>, but I think it works the same no matter which version I used, each time when the train function is called, so for each batch the gradient is set to zero.</p>
<p>I made some changes, now I accumulate decoder outputs, and then calculate loss on entire batch.</p>
<pre><code> for input_seq, target_seq, input_lengths, target_lengths in train_loader:
    # Run a training iteration with batch

    output = train(input_seq, target_seq, input_lengths, target_lengths, encoder, decoder, embedding,
      encoder_optimizer, decoder_optimizer, clip, teacher_forcing_ratio, criterion)
    
    #print(output.contiguous().view(-1, len(voc.word_count)).shape)
    #print(target_seq.contiguous().view(-1).shape)

    loss = criterion(output.view(-1, len(voc.word_count)), 
                     target_seq.contiguous().view(-1))
    loss = loss / target_seq.size(0)

    loss.backward()          
    encoder_optimizer.step()
    decoder_optimizer.step()
    encoder_optimizer.zero_grad()
    decoder_optimizer.zero_grad()
</code></pre>
<p>`<br>
In train function loop through words in given sequences:</p>
<pre><code> #Forward propagate through decoder using SOS tokens
decoder_output, decoder_hidden = decoder(decoder_input, decoder_hidden, encoder_outputs)

output_vectors.append(decoder_output)

#Forward batch of sequences through decoder one time step at a time
if use_teacher_forcing:
    for t in range(seq_len-1):
        
        # Teacher forcing: current target word is the next input
        decoder_input = target_seq[:, t].view(-1,1) # (batch_size, 1)
        
        # Forward propagate through decoder
        decoder_output, decoder_hidden = decoder(decoder_input, decoder_hidden, encoder_outputs)
        
        output_vectors.append(decoder_output)
 return torch.cat(output_vectors, dim=1)
</code></pre>
<p>`<br>
When calculation loss decoders inputs correspond to targets in following manner:<br>
decoder_input (train on) SOS-&gt;1-&gt;2-&gt;3-&gt;4<br>
targets                                   1-&gt;2-&gt;3-&gt;4-&gt;END</p>
<p>In loss function I used following shapes:<br>
loss = criterion(output.view(-1, len(voc.word_count)),<br>
target_seq.contiguous().view(-1))<br>
output shape: [batch_size * seq_len, vocab_size]<br>
target shape: [batch_size * seq_len]</p>
<p>It is totally different approach, but loss still fluctuates at the same level.</p>
<p>Any ideas what can be wrong here?</p>
<p>Best regards.<br>
Radek</p>

kedar: <p>I checked out the sum of the gradients of <code>encoder.embedding.weight.grad</code> during training for model that uses mask padding and for second one that uses criterion with ignore_index.<br>
For model with padding mask the lower the loss was, the lower the gradients, but they decreased very slowly.<br>
Iteration: 1; tensor(-0.1997)<br>
Iteration: 35; tensor(-0.0057)</p>
<p>In contrast to preceding example, model with <code>criterion</code> begins with much lower gradients, and during the training gradients diminish much faster.<br>
Iteration: 1; tensor(-5.2681e-05)<br>
Iteration: 35; tensor(2.0964e-10)</p>
<p>The very small gradients cause that model doesn’t converge, but how to find a remedy to that problem? Could anybody please explain gradients behavior? Any other advice would be appreciated as well.</p>

kedar: <p>I figured out what was wrong with my model. It turned out that despite that my loss function returned some reasonable values, the loss was not calculated properly, thus as a consequence model did not learn. Output from my AttentionDecoder was softmaxed, then I used CrossEntropyLoss or NLLLoss (tried them both), but I did not change the softmax to log_softmax in case of NLLLoss, or in case of using CrossEntropyLoss I did not get rid of softmax at all as CEL comprised of log_softmax and NLLLoss.</p>