What exactly does `retain_variables=True` in `loss.backward()` do?

dl4daniel: <p>In the doc it says:</p>
<pre><code>retain_variables (bool): If ``True``, buffers necessary for computi
 133                     gradients won't be freed after use. It is only necessary to
 134                     specify ``True`` if you want to differentiate some subgraph mul
 135                     times (in some cases it will be much more efficient to use
 136                     `autograd.backward`).
</code></pre>
<p>one way to understand it is to “keep all variables or flags associated with computing gradients” while <code>retain_variables=False</code> make values of variables or flags gone.</p>
<p>However, I am not sure I understand it properly, I would like to see the differences in codes and outputs. So, I tried to look into the source code, and here is as far as I can go below:</p>
<pre><code>def backward(self, gradient=None, retain_variables=False):
 117             """Computes the gradient of current variable w.r.t. graph leaves.
 118
 119             The graph is differentiated using the chain rule. If the variable is
 120             non-scalar (i.e. its data has more than one element) and requires
 121             gradient, the function additionaly requires specifying ``gradient``.
 122             It should be a tensor of matching type and location, that contains
 123             the gradient of the differentiated function w.r.t. ``self``.
 124
 125             This function accumulates gradients in the leaves - you might need to z
 126             them before calling it.
 127
 128             Arguments:
 129                 gradient (Tensor): Gradient of the differentiated function
 130                     w.r.t. the data. Required only if the data has more than one
 131                     element. Type and location should match these of ``self.data``.
 132                 retain_variables (bool): If ``True``, buffers necessary for computi
 133                     gradients won't be freed after use. It is only necessary to
 134                     specify ``True`` if you want to differentiate some subgraph mul
 135                     times (in some cases it will be much more efficient to use
 136                     `autograd.backward`).
 137             """
 138             if self.volatile:
 139                 raise RuntimeError('calling backward on a volatile variable')
 140             if gradient is None and self.requires_grad:
 141                 if self.data.numel() != 1:
 142                     raise RuntimeError(
 143                         'backward should be called only on a scalar (i.e. 1-element
 144                         'or with gradient w.r.t. the variable')
 145                 gradient = self.data.new().resize_as_(self.data).fill_(1)
 146  -&gt;         self._execution_engine.run_backward((self,), (gradient,), retain_variab
 return None
</code></pre>
<p>Apparently, to see what exact does <code>retain_variables</code> do in <code>run_backward</code>, I have to at least a level deeper, but <code>step</code> in <code>pdb</code> won’t take me there, it just <code>return None</code>. So, I am stuck.</p>
<p>could anyone help me here? Thanks</p>

mratsim: <p>After <code>loss.backward</code> you cannot do another <code>loss.backward</code> unless <code>retain_variables</code> is true.</p>
<p>In plain words, the backward proc will consume the intermediate saved Tensors (Variables) used for backpropagation unless you explicitly tell PyTorch to retain them.</p>

dl4daniel: <p>Thanks a lot! Your explanation is helpful.</p>
<p>I still wonder how can I use some code to experiment on the differences made with and without <code>retain_vairables</code>.</p>
<p>Could you give me some simple code examples on showing the differences?</p>
<p>or could you show me how to access the source code of <code>self._execution_engine.run_backward</code>?</p>
<p>Thanks a lot!</p>

mratsim: <p>Usually after a backpropagation you process the next batch so you don’t need the gradients of the previous batch anymore.</p>
<p>Besides debugging I don’t have any scenario for which I need to backprop twice through the same operation graph, I’m sure there are though.</p>
<p>The default is to drop the variables/gradient to save on memory.</p>
<p>All the source code of PyTorch is on github.  From the name, I guess that <code>self._execution_engine.run_backward</code> is traversing the operation graph in reverse, calling <code>backward</code> if defined, or using the <code>autograd</code> if not.</p>
<p>Each node in the graph have several properties that are defined in the <code>autograd</code> folder of PyTorch</p>

11117: <p>Actually, when we train a GAN, we usually should calculate the D_loss for discrimination net and G_loss for generator net And after calculating the gradient of D_loss,we need to retain variables for the calculation of G_loss,which is a typical example the “retain_variables=True” use.</p>

tom: <p>In my (very limited) experience, you do not compute discriminator and generator gradients for the same forward step because the loss objectives have opposing signs.</p>
<p>Best regards</p>
<p>Thomas</p>

11117: <p>Thank you for reply, I now think my example has a little inaccuracy, and you are right.</p>
<p>Thanks for your correction!</p>

jdhao: <p>I think a concrete case where <code>retain_graph=True</code> is helpful is multi-task learning where you have different losses at different layers of the network. So in order to back-propagate the gradient of each loss w.r.t to the parameters of the network, you will need to set <code>retain_graph=True</code>, or you can only do backward for one of the many losses.</p>

yoelshoshan: <p>Another example where it is useful is shown here:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pkdn/pytorch-smoothgrad/blob/master/lib/gradients.py#L79" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pkdn/pytorch-smoothgrad/blob/master/lib/gradients.py#L79" target="_blank" rel="nofollow noopener">pkdn/pytorch-smoothgrad/blob/master/lib/gradients.py#L79</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="69" style="counter-reset: li-counter 68 ;">
<li>    one_hot = np.zeros((1, output.size()[-1]), dtype=np.float32)</li>
<li>    one_hot[0][index] = 1</li>
<li>    if self.cuda:</li>
<li>        one_hot = Variable(torch.from_numpy(one_hot).cuda(), requires_grad=True)</li>
<li>    else:</li>
<li>        one_hot = Variable(torch.from_numpy(one_hot), requires_grad=True)</li>
<li>    one_hot = torch.sum(one_hot * output)</li>
<li>
</li>
<li>    if x_plus_noise.grad is not None:</li>
<li>        x_plus_noise.grad.data.zero_()</li>
<li class="selected">    one_hot.backward(retain_variables=True)</li>
<li>
</li>
<li>    grad = x_plus_noise.grad.data.cpu().numpy()</li>
<li>
</li>
<li>    if self.magnitutde:</li>
<li>        total_gradients += (grad * grad)</li>
<li>    else:</li>
<li>        total_gradients += grad</li>
<li>    #if self.visdom:</li>
<li>
</li>
<li>avg_gradients = total_gradients[0, :, :, :] / self.n_samples</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Here the input image is treated as variables, and a gradient is calculated to see which parts in the image will have most influence on the global classification decision.<br>
This is useful when you trained a classifier without any localization and you still want to see some localization information.</p>

jpeg729: <p>The question asks about <code>retain_variables</code>, some answers talk about <code>retain_graph</code>. What is the difference between the two?</p>

jdhao: <p>It is essentially the same, <code>retain_variables</code> argument has been deprecated in favor of <code>retain_graph</code>.</p>

devansh20la: <p>I understand If I have two loss functions in different parts of the network, I’ll have use retain_graph. What if I add both the losses and do total_loss.backward() ?</p>
<p>for example:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/aace8d6020dc8652caf356257db5cc06fe793d42.JPG" data-download-href="https://discuss.pytorch.org/uploads/default/aace8d6020dc8652caf356257db5cc06fe793d42" title="IMG_6691"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aace8d6020dc8652caf356257db5cc06fe793d42_2_690x204.JPG" alt="IMG_6691" data-base62-sha1="on1HMlNouIvj6W1Ty3gJfAHvDG2" width="690" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aace8d6020dc8652caf356257db5cc06fe793d42_2_690x204.JPG, https://discuss.pytorch.org/uploads/default/optimized/2X/a/aace8d6020dc8652caf356257db5cc06fe793d42_2_1035x306.JPG 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/a/aace8d6020dc8652caf356257db5cc06fe793d42_2_1380x408.JPG 2x" data-dominant-color="B4B1AB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_6691</span><span class="informations">3395×1008 532 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Rather than</p>
<pre><code class="lang-auto">loss1.backward(retain_graph=True)
loss2.backward()
opt.step()
</code></pre>
<p>I would just do</p>
<pre><code class="lang-auto">total_loss = loss1 + loss2
total_loss.backward()
opt.step()
</code></pre>

smth: <p><a class="mention" href="/u/devansh20la">@devansh20la</a> that is pretty legit and works well.</p>

PetrusKe: <p>Hi, <a class="mention" href="/u/smth">@smth</a> ,<br>
I just started to use pytorch recently. And I also comfuse about above problem.<br>
And I think above two ways may not same?</p>
<p>While use:<br>
loss1.backward(retain_graph=True)<br>
_ loss2.backward()_<br>
_ opt.step()_<br>
the layers between loss1 and loss2 will only calculate gradients from loss2.<br>
and the layers before loss1 will calculate gradientes as sum of loss1+loss2</p>
<p>but if use:<br>
total_loss = loss1 + loss2<br>
_ total_loss.backward()_<br>
_ opt.step()_<br>
all layers will calculate gradient by using loss_value = loss1 + loss2</p>
<p>What do you think about it?</p>

devansh20la: <aside class="quote no-group" data-username="PetrusKe" data-post="15" data-topic="3508">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/a8b319/48.png" class="avatar"> PetrusKe:</div>
<blockquote>
<p>he layers between loss</p>
</blockquote>
</aside>
<p>derivative of weights of model 2 wrt to loss 1 would be zero.</p>

STU: <p>‘loss1.backward(retain_graph=True)’ will update the model-1’s parameters,<br>
when we do loss2.backward(), will update the model-1?</p>

STU: <p>we need the variables or the part of computational graph of model-1, but we won’t update it’s parameters? e.g. in GAN. Are there any mistakes with my understanding?</p>

devansh20la: <p>Yes, it will update model1 parameters. Gradients from loss2 will flow all the way to model1.</p>

Insight_Gong: <p>So what’s the difference between these 2 methods?<br>
I don’t know how the <code>loss1+loss2</code> is trying to backward.<br>
Do they have diff? What’s the diff?</p>

Insight_Gong: <p>As far as I think, loss = loss1 + loss2 will compute grads for all params, for params used in both <code>l1</code> and <code>l2</code>, it sum the grads, then using backward() to get grad.<br>
Meanwhile, loss1.backward() and loss2.backward() means seperately compute grads in loss1 and loss2, and get grads themselves.</p>
<p>The diff is that: When using optimizer to update step size using <code>gd</code> method, sum(loss).backward() using only 1 optimizer with <code>sum</code> of grads; and <code>l1, l2</code> using 1 optimizer(or even 2 optimizer) to compute step-size based on the grads, for params which l1 and l2 owns in public, maybe the step-size is depended on <code>l2</code> loss(which computes later).</p>