Backpropagation through the training procedure

dbp.pat94: <p>I am trying to replicate something that the authors in the paper, <a href="https://arxiv.org/pdf/1706.05394.pdf" rel="noopener nofollow ugc">A Closer Look at Memorization in Deep Networks</a>, have proposed in the <em>Section 3.2: Loss-Sensitivity in Real vs. Random Data</em> which says:</p>
<blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85d7f6a1de4181ea1988ee7cb7f45e1ea4b97707.png" data-download-href="https://discuss.pytorch.org/uploads/default/85d7f6a1de4181ea1988ee7cb7f45e1ea4b97707" title="mem-loss-sen-ques"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85d7f6a1de4181ea1988ee7cb7f45e1ea4b97707.png" alt="mem-loss-sen-ques" data-base62-sha1="j62cAtcuIITHuyKDb8fnCIEVpyL" width="439" height="500" data-dominant-color="E5E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mem-loss-sen-ques</span><span class="informations">501×570 80.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</blockquote>
<p>It’s not clear from the discussion in <a href="https://arxiv.org/pdf/1502.03492.pdf" rel="noopener nofollow ugc"><strong>Maclaurin et al. (2015)</strong> </a> or <a href="https://github.com/HIPS/hypergrad/" rel="noopener nofollow ugc">their github code</a> as to what <em>I</em> want to do which is essentially to <strong><em>backprop through training procedure</em></strong>.</p>
<p><strong>A Digression (?):</strong> Although I am not quite familiar with Recurrent Neural Networks (RNNs) and haven’t used them in my work, I think it looks a bit similar to (but not exactly the same as) Truncated Backpropagation Through Time (TBPTT). I went through the <a href="https://discuss.pytorch.org/t/implementing-truncated-backpropagation-through-time/15500/3">discussion here</a> but I am still not sure (in case it is indeed similar to TBPTT) how to implement what I need and have mentioned above. For instance, what changes would I need to make here:</p>
<pre><code class="lang-auto"># perform k1-step unrolling through the training procedure

def train(data_loader, optimizer, model):
   model.train()
   cnt = 0
   # store the loss sensitivity values in a Tensor
   loss_sensitivity = torch.zeros((len(data_loader.dataset), data_loader.tensors[0].shape[1], num_classes))

   idx_last_k1 = [] # keeps track of indices of (K-k1)th batch where K is the current grad. step
   for batch_id, (x,y, idx) in enumerate(data_loader):

      idx_last_k1 = '''store (K-k1)th batch indices where K is the current step'''
      x.requires_grad = True

      if batch_id % k1 == 0:
         # store the corresponding loss sensitivity values
         loss_sensitivity[idx,:] = data_loader.tensors[0][idx_last_k1,:].grad.clone().detach()

         # We compute gradients (for sensitivity) 
         # only w.r.t. the (M-k1)th batch
         data_loader.tensors[0][idx_last_k1, :].detach_()
         idx_last_k1 = []
      ...
      # some more usual training operations
      ...
      output = model(x)
      loss = cce_loss(output, y)
      optimizer.zero_grad()
 
      loss.backward(retain_graph = True)
      optimizer.step()
      ...
</code></pre>
<p><a class="mention" href="/u/alband">@albanD</a> Any idea about this?</p>

dbp.pat94: <p><a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> Any ideas? I am stuck.</p>
<p>One thing that I realise is missing from the code snippet I have shared is that there’s no provision for storage of gradients of the form: <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07ba991e2e2f085faf84fd7399f1899cd079f30f.png" alt="part_theta" data-base62-sha1="16n7RRhItY6p2IlhRPhhLsUZ8LR" width="59" height="59"> owing to the fact that <code>nn.Parameters</code> are <code>leaf Tensors</code> because of which any operations on them won’t be traced in the computation graph.</p>
<p>For that, I think I will have to <a href="https://discuss.pytorch.org/t/higher-order-derivatives-meta-learning/93051">employ the hack</a> needed for meta-learning wherein we need these kinds of derivatives. But even with that, I still haven’t been able to answer my question completely.</p>

albanD: <p>Hi,</p>
<p>You might want to take a look at <a href="https://github.com/facebookresearch/higher/">https://github.com/facebookresearch/higher/</a> that gives a lot of utilities to perform these tasks!</p>

dbp.pat94: <p>Thanks for responding to this, <a class="mention" href="/u/alband">@albanD</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> However, that’s not my main concern at the moment. I wish to first understand what they really mean by <code>unrolling the SGD steps and backpropagating through this unrolled computation graph</code>. Can you help me understand and unpack what’s going on here?</p>

albanD: <p>From what I understand is that when you do regular ML, you do something like this after unrolling the training loop:</p>
<pre><code class="lang-auto"># Iteration 1
loss = f(w0)
g = grad(loss, w0)
w1 = w0 - lr * g
# Iteration 2
loss = f(w1)
g = grad(loss, w1)
w2 = w1 - lr * g
# Iteration 3
loss = f(w2)
g = grad(loss, w2)
w3 = w2 - lr * g
</code></pre>
<p>So you only ever backprop to the latest w value.<br>
In this case, you can actually ask for gradients for older “versions” of w like for the loss at iteration 3, you can do <code>grad(loss, w0)</code>. And in that case, you can see that the gradient update by SGD is actually part of the differentiable operations that the autograd need to differentiate through.</p>

dbp.pat94: <p>So, in that case, what I have mentioned about requiring <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07ba991e2e2f085faf84fd7399f1899cd079f30f.png" alt="part_theta" data-base62-sha1="16n7RRhItY6p2IlhRPhhLsUZ8LR" width="59" height="59"> is correct, right? In which case I either need to use <code>higher</code> package as you mentioned or the other hack that I mentioned above I suppose.</p>
<p>But other than that, does the code snippet that I have put up make sense? My main confusion is about whether or not I have used <code>retain_graph=True</code> correctly. So, for instance, since I want to compute the <code>loss sensitivity</code> every <code>k1</code> steps, I am <code>detaching</code> the older <code>x</code>’s and keeping <code>requires_grad=True</code> only for the last <code>k1 x</code>’s.</p>

albanD: <p>It does look like what you want is very similar to truncated backpropagation through time. Only that in your case, the evaluations are not timestep but optimization steps.<br>
You can check this post that gives an example of how to do it manually: <a href="https://discuss.pytorch.org/t/implementing-truncated-backpropagation-through-time/15500/4" class="inline-onebox">Implementing Truncated Backpropagation Through Time - #4 by albanD</a></p>
<blockquote>
<p>data_loader.tensors[0][idx_last_k1, :].detach_()</p>
</blockquote>
<p>I am not sure what you’re trying to achieve here, but it most likely doesn’t work <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20">  The indexing of [idx_last_k1, :] is returning a temporary view on the original tensor. And then you detach that temporary view inplace. But that does not modify the original Tensor!</p>
<blockquote>
<p>whether or not I have used <code>retain_graph=True</code> correctly</p>
</blockquote>
<p>You should only ever use it if you compute gradients on the same graph multiple times. Otherwise, there is something wrong.</p>

dbp.pat94: <blockquote>
<p>But that does not modify the original Tensor!</p>
</blockquote>
<p>Oh! I didn’t know that indexing like this returns a view. Then how do I detach those original <code>Tensors</code>? What modification do I need to make to detach only those particular <code>x</code>’s?</p>
<blockquote>
<p>You should only ever use it if you compute gradients on the same graph multiple times. Otherwise, there is something wrong.</p>
</blockquote>
<p>This is a slightly rhetorical question but let me ask it anyway: But I do need to retain the derivatives, right? I need derivatives up to <code>k</code> steps in the ‘past’.</p>

dbp.pat94: <p>Hi, <a class="mention" href="/u/alband">@albanD</a>, sorry for pestering you for so long, but I didn’t understand how to do the following:</p>
<blockquote>
<p>How does one detach so that it modifies the original Tensor as well and not just the <code>view</code>?</p>
</blockquote>
<p>I am talking specifically about the code snippet I have written:</p>
<pre><code class="lang-auto"># some code
...
...
data_loader.tensors[0][idx_last_k1, :].detach_()
...
...
# some code
</code></pre>
<p>How do I detach those particular <code>Tensors</code> selected by <code>idx_last_k1</code>specifically because as per what you said, with this code, I am only detaching the temporary views in-place and <strong>not</strong> the original <code>Tensors</code>. The reason why I am asking this is that I still need to be able to understand and do this even if I were to use packages for higher-order derivatives (such as <code>higher</code>) for computing things like <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07ba991e2e2f085faf84fd7399f1899cd079f30f.png" alt="part_theta" data-base62-sha1="16n7RRhItY6p2IlhRPhhLsUZ8LR" width="59" height="59"> .</p>

albanD: <p>Hi,</p>
<p>Sorry for the delay in the answer, that thread dropped out of my unread list…</p>
<blockquote>
<p>How does one detach so that it modifies the original Tensor as well and not just the view?</p>
</blockquote>
<p>The detach/not detach is a property of a Tensor. So if you don’t call it on the original Tensor, it will have no effect.<br>
You can check the thread I linked above that show how to do it.<br>
Also note that this is not retro-active: you need to detach() (or detach_() which is just a shorthand for <code>x = x.detach()</code>) before using the Tensor, otherwise it won’t have any effect.</p>
<blockquote>
<p>But I do need to retain the derivatives, right? I need derivatives up to <code>k</code> steps in the ‘past’.</p>
</blockquote>
<p>If you plan to backprop in that same graph again, then yes!<br>
But in your current code it is not clear if you were planning to do that or not so I just mentioned it as it is a common error people do.</p>

dbp.pat94: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>I went through <a href="https://discuss.pytorch.org/t/implementing-truncated-backpropagation-through-time/15500/4">Implementing Truncated Backpropagation Through Time</a> as you suggested but I think I am a little confused about this</p>
<blockquote>
<p>this is not retro-active: you need to detach() (or detach_() which is just a shorthand for <code>x = x.detach()</code> ) before using the Tensor, otherwise it won’t have any effect.</p>
</blockquote>
<p>So, now I will put my question a little differently and see if it makes sense:</p>
<blockquote>
<p>How do I detach only certain indices/slices of the original Tensor. The reason why I am asking this is because I, at a given training iteration, only want to track gradient updates corresponding to the last <code>k1</code> batches of inputs (<code>x</code>’s). So, I am interested in keeping <code>requires_grad=True</code> only for those inputs (<code>x</code>) and <code>requires_grad=False</code> for the rest of the inputs.</p>
</blockquote>

albanD: <aside class="quote no-group" data-username="dbp.pat94" data-post="11" data-topic="103813">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dbp.pat94/48/27134_2.png" class="avatar"> dbp.pat94:</div>
<blockquote>
<p>How do I detach only certain indices/slices of the original Tensor.</p>
</blockquote>
</aside>
<p>You cannot do that I am afraid. The requires_grad property is only for a full Tensor and cannot be set for a subset of it only.</p>
<p>If you put them in a list though, you can set these attributes on each Tensor as needed.</p>