Correct way to do backpropagation through time?

username: seanny1986
cooked: <p>This is in the context of model-based reinforcement learning. Say I have some reward at time T, and i want to do truncated backprop through the network roll out, what is the best way to do this? Are there any good examples out there? I haven’t managed to find much.</p>
<p>Any help would be appreciated!</p>

username: smth
cooked: <pre><code class="lang-auto"># non-truncated
for t in range(T):
   out = model(out)
out.backward()

# truncated to the last K timesteps
for t in range(T):
    out = model(out)
    if T - t == K:
        out.detach()
out.backward()
</code></pre>

username: jpeg729
cooked: <p>Shouldn’t the truncated example be</p>
<pre><code># truncated to the last K timesteps
for t in range(T):
    out = model(out)
    if T - t == K:
        out.backward()
        out.detach()
out.backward()</code></pre>

username: smth
cooked: <p>yea you’re right. i was just backwarding through the last part</p>

username: Santosh_Manicka
cooked: <p>I guess that an alternative is to do a “head” truncation besides a “tail” trunction</p>
<p>This is tail truncation –</p>
<aside class="quote no-group" data-username="smth" data-post="2" data-topic="11701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>for t in range(T):<br>
out = model(out)<br>
if T - t == K:<br>
out.detach()<br>
out.backward()</p>
</blockquote>
</aside>
<p>This is head truncation –</p>
<pre><code class="lang-auto">modelparameter.requires_grad = False
for t in range(T):
    out = model(out)
    if T - t == K:
        modelparameter.requires_grad = True
out.backward()
</code></pre>

username: riccardosamperna
cooked: <p>I have a similar issue in this <a href="https://discuss.pytorch.org/t/neural-style-transfer-on-videos/22650/5">post</a>.</p>
<p>I followed the pseudocode for the non-truncated BPTT in this conversation, the network trains but I have the feeling that the gradient is not flowing through time. I posted my training code for the network.</p>
<p>Can someone give some tips?</p>

username: DuaneNielsen
cooked: <p>Check out hooks.  If you want to inspect an gradient, you can register a backwards_hook, and drop the values into a print statement or tensorboard.</p>
<aside class="quote quote-modified" data-post="2" data-topic="94">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/2">Why cant I see .grad of an intermediate variable?</a> 
  </div>
  <blockquote>
    Hi Kalamaya, 
By default, gradients are only retained for leaf variables. non-leaf variables’ gradients are not retained to be inspected later. This was done by design, to save memory. 
However, you can inspect and extract the gradients of the intermediate variables via hooks. 
You can register a function on a Variable that will be called when the backward of the variable is being processed. 
More documentation on hooks is here: <a href="http://pytorch.org/docs/autograd.html#torch.autograd.Variable.register_hook" rel="noopener nofollow ugc">http://pytorch.org/docs/autograd.html#torch.autograd.Variable.regis…</a>
  </blockquote>
</aside>

<p>eg, in the below code I drop a hook to monitor the values passing through a softmax functiion.  (later I compute the entropy and pump it into tensorboard).</p>
<pre><code class="lang-auto">        def monitorAttention(self, input, output):
            if writer.global_step % 10 == 0:
                monitors.monitorSoftmax(self, input, output, ' input ', writer, dim=1)
        self.softmax.register_forward_hook(monitorAttention)
</code></pre>

username: Ponc_Palau_Puigdeval
cooked: <p><a class="mention" href="/u/riccardosamperna">@riccardosamperna</a> did you find out the correct way to solve this problem? I followed the 2 posts you mentioned but I found no solution</p>

username: dbp.pat94
cooked: <p>Does <a href="https://discuss.pytorch.org/t/implementing-truncated-backpropagation-through-time/15500">this discussion</a> help?</p>