Clarification on Backpropagation for LSTMs

eth_er: <p>I’m trying to train an image captioning model using 2 LSTMs, however, I’m confused with how to properly calculate the loss and make the network learn. The tutorials I see for LSTMs (not necessarily for image captioning) seem to pass in a whole sentence and instantly get back the loss for the whole sentence (as in <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html</a> POS tagging).</p>
<p>For clarification <span class="hashtag">#1:</span><br>
I was under the impression that you have to pass each word to the LSTM timestep-by-timestep (which means there’s an inner for loop that iterates through each word), and each time, you have to pass in the previous word and the previous hidden state to the LSTM. Is this correct?</p>
<p>For clarification <span class="hashtag">#2:</span><br>
If <span class="hashtag">#1</span> is correct, then, at each timestep, you get an output, and you incur a loss (for example, cross-entropy). To make the LSTM learn, do I (1) immediately call <code>loss.backward()</code> but run <code>optimizer.step()</code> after the whole sequence, or (2) accumulate the loss for all the timesteps, and only call <code>accumulated_loss.backward()</code> after the sequence (followed of course by <code>optimizer.step()</code>)?</p>
<p>Would really appreciate the help. I’ve trained my network using method (2) but it doesn’t go lower than a certain point. I also tried running it on a small data set to make it overfit and preferrably memorize the training data just to see if it learns sequences, but the outputs don’t show any sentence structure. The highly probable words are almost always the same for all of the timesteps.</p>
<p>Thanks in advance!</p>

vdw: <p>Re <span class="hashtag">#1:</span> <code>LSTM</code> takes the whole sequence and performs each time step in the background. However, nothing is stopping you give <code>LSTM</code> just one word at a time. It depends on your task and how you want to implement it.</p>
<p>Re <span class="hashtag">#2:</span> I think (1) is not correct since you backpropagate multiple times over the same past time steps. (2) is the common approach for encoder-decoder models (e.g., autoencoder, machine translation) where the decoder generate words step by step. Sequence tagging is actually a bit more straightforward. If you look at your linked POS tagging tutorial, there is not loop needed. <code>tag_scores = model(sentence_in)</code> contains the output of all time steps. So you can calculate the loss in one go with <code>loss = loss_function(tag_scores, targets)</code></p>
<p>Given that you want to do image captioning (i.e., generating text), you need a loop and do approach (2): accumulating the loss and call <code>loss.backward()</code> only once. Have a look at the <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">Sequence-to-Sequence tutorial</a>, particularly at the decoder part. Your decoder will roughly look the same. The only difference is that the hidden state won’t come from a text encoder but from some image encoder.</p>
<p>I hope that helps.</p>

eth_er: <p>Thanks so much for taking the time! This was helpful. I just have two more questions:</p>
<ol>
<li>
<p>The <code>LSTM</code> that you were talking about that performs each time step in the background, specifically, is the <code>nn.LSTM</code>, not <code>nn.LSTMCell</code>, right? This might have been one of my confusions.</p>
</li>
<li>
<p>If training the LSTM by batch, I pad the captions to have the same length, and the final loss is the mean of the loss of all the targets (excluding padded ones). Should I be doing anything else?</p>
</li>
</ol>

vdw: <p>Re <span class="hashtag">#1:</span> No, I was talking solely about <code>LSTM</code>. It wraps an <code>LSTMCell</code> to support multiple layers, dropout, bidirectionality, etc. An <code>LSTMCell</code> only ever takes just one word as input. <code>LSTM</code> is a full layer allowing for whole sequences as output. It’s just that no-one is stoping you to give it sequences of length 1. An <code>LSTM</code> with <code>num_layers=1</code>, <code>bidirectional=False</code> and <code>dropout=0.0</code>that takes one word at a time should be more or less the same as an <code>LSTMCell</code>. My recommendation, stick with <code>LSTM</code> for the time being, and consider <code>LSTMCell</code> if you really need more control of the recursion.</p>
<p>Re <span class="hashtag">#2:</span> Yes, padding is the common technique. Don’t forget to at a fixed EOS (end-of-sequence) token behind each question, so the decoder learns when to stop generating a caption. At the beginning, you can just use padding and see how it works. For more advanced methods, you can check <code>PackedSequence</code> or even sort your dataset in such a way, that each batch has captions of the same length.</p>
<p>In short: Use <code>LSTM</code> and padding to get a basic network training. Then you can see if you want and can improve.</p>