Should we .detach() predicted model outputs used as input in seq2seq model training?

Messiah: <p>When training an RNN model for seq2seq tasks, we can incorporate a decision to use either the predicted model output or the ground-truth target as input for the current batch or timestep. Should we <code>.detach()</code> the predicted model output in the former case?</p>
<p>The source of my confusion arises from an apparent discrepency in two PyTorch tutorials using a similar encoder-decoder architecture for seq2seq tasks:<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html </a></p>
<pre><code class="lang-auto">decoder_input = topi.squeeze().detach()  # detach from history as input
</code></pre>
<p><a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/chatbot_tutorial.html </a></p>
<pre><code class="lang-auto">decoder_input = torch.LongTensor([[topi[i][0] for i in range(batch_size)]])
</code></pre>
<p>A related concept is called “Scheduled Sampling” (<a href="https://arxiv.org/pdf/1506.03099.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1506.03099.pdf</a>), and in this approach they also do not backpropogate through the model predicted outputs.</p>
<p>In the disussion <a href="https://stackoverflow.com/questions/61187520/should-decoder-prediction-be-detached-in-pytorch-training" class="inline-onebox" rel="noopener nofollow ugc">python - Should Decoder Prediction Be Detached in PyTorch Training? - Stack Overflow</a> it is mentioned that we do want to <code>.detach()</code> due to the extra memory usage of keeping those variables in the computation graph. I am wondering however how this relates to the overall performance of the model in terms of accuracy and not efficiency.</p>
<p>I would like to know why we would want to <code>.detach()</code> the model predicted outputs from the computation graph, and in what contexts we may not want to do that. If we are using an RNN model, doesn’t this mean that we are breaking BPTT which RNN relies on to learn effectively? I am dealing with a similar situation where I need to iterate through each timestep of variable-length batched sequences during training, but I am unsure as to whether I should <code>.detach()</code>  the model predicted outputs used as input during those training iterations where these inputs are used. I intuit that in some cases we would indeed want to backpropogate through model predicted outputs during training.</p>

ptrblck: <p>Both approaches will create a detached tensor since rewrapping a tensor also does not keep the gradient history alive.</p>
<p>If you are not detaching the input you will keep the gradient history alive, which will allow you to calculate the gradients w.r.t. all used parameters of all iterations, but will also increase the memory usage (since the intermediate tensors of all iterations are kept alive) and could also yield to errors.<br>
Errors could be raised if you have already updated the used parameters via <code>optmizer.step()</code> will the computation graph is still attached to all iterations.<br>
Here is a small example:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 10)
)
optimizer = torch.optim.SGD(model.parameters(), lr=1.)

# works since we are creating one lage computation graph from 10 iterations
x = torch.randn(1, 10)
for _ in range(10):
    x = model(x)

x.mean().backward()
optimizer.step()

# breaks
optimizer.zero_grad()
x = torch.randn(1, 10)
for i in range(10):
    print(i)
    x = model(x)
    x.mean().backward(retain_graph=True)
    optimizer.step()

# output
# 0
# 1
# RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10, 10]], which is output 0 of AsStridedBackward0, is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>This error is raised since the intermediate forward activations from the first iterations are “stale” and were not created by the current parameters of the model after calling <code>optimizer.step()</code>.</p>

srishti-git1110: <aside class="quote no-group" data-username="Messiah" data-post="1" data-topic="174750">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/3e96dc/48.png" class="avatar"> Messiah Ataey:</div>
<blockquote>
<p>I would like to know why we would want to <code>.detach()</code> the model predicted outputs from the computation graph, and in what contexts we may not want to do that. If we are using an RNN model, doesn’t this mean that we are breaking BPTT which RNN relies on to learn effectively?</p>
</blockquote>
</aside>
<p>The predicted model outputs are essentially used as inputs to the next time steps. We ideally do not have any computation graph attached with input tensors and so, they are generally detached.</p>
<p>With teacher forcing, the ground truths do not come with a graph history attached to them, so this is also in sync with the ideal case.</p>
<p>This also explains why BPTT isn’t broken with detached tensors.</p>
<p>As <a class="mention" href="/u/ptrblck">@ptrblck</a> has explained, it is possible to experiment with a case where the predicted outputs are used as next inputs without detaching.<br>
I am not sure what performance difference it will bring in terms of the accuracy for any kind of auto-regressive model, but generally detaching is done to mimic the ideal case.</p>
<p>It would nevertheless be interesting to see if un-detached inputs have been experimentally explored with auto-regressive models.</p>

Messiah: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a> for the clarification! It makes sense why we should <code>detach()</code> those variables since we usually do not want to associate gradients with inputs. I seem to have confused the output of the GRU/LSTM with the hidden state, which in fact is backpropogated through (and in the case of per timestep training is actually the same as the <code>output</code> of a GRU/LSTM at each timestep).</p>
<p>I would like to ask a few other questions if you don’t mind:</p>
<ol>
<li>
<p>Is BPTT executed on an RNN model when we call <code>loss.backwards()</code>, where <code>loss</code> is say the accumulated loss across each timestep of a training sequence, after which we call <code>optim.step()</code>, which updates model parameters the gradients calculated using all prior outputs (hidden states) of the sequence?</p>
</li>
<li>
<p>What if we call <code>loss.backward(retain_graph=True)</code> and <code>optim.step()</code> at <em>each</em> timestep, but <code>detach()</code> the hidden state (in order to avoid the error as you show)? There is no BPTT happening in this case, since we break the computation graph at each timestep. Is there then a way to update model parameters at each timestep of a sequence without needing to <code>detach()</code> the hidden state? It seems this issue was raised <a href="https://discuss.pytorch.org/t/how-to-backpropagate-a-loss-through-time-series-rnn/151327/6">before</a> but was left unanswered.</p>
</li>
<li>
<p>When training a LSTM/GRU, there are normally two inputs provided to the model at each timestep: the current state features and the previous hidden state. Training seq2seq models often uses ‘teacher forcing’, where the ground truth targets are provided as the current state input at each timestep as opposed to the prior model prediction. How does BPTT differ when training a model using teacher forcing and when using its own model predictions (i.e., autoregressive training), and what specific autograd considerations should be made in the latter approach if any?</p>
</li>
</ol>