Quick Detach() Question

username: ajbrock
cooked: <p>If I call detach() on a variable after calling backward(retain_variables=True) on a variable further along the graph, will that free the buffers of the graph that led to the detach()'d variable?</p>
<p>Example: net1 and net2 are modules, x is an input variable. I want to retain the state of net2 for processing later down the line, but I just want to accumulate gradients in net1 for the time being, no need to retain the graph.</p>
<p>out1 = net1(x)</p>
<p>out2 = net2(out1)</p>
<p>loss = some_loss_function(out2)</p>
<p>loss.backward(retain_variables=True)</p>
<p>out1.detach_()</p>
<p>Will this clear the buffers of net1?</p>
<p>Thanks,</p>
<p>Andy</p>

username: albanD
cooked: <p>Hi,</p>
<p>From my understanding it won’t.<br>
One way to do this (maybe not the best) would be to separate these in two graphs and backpropagate the second one with <code>retain_variables</code> and then backpropagate the first one without:</p>
<pre><code class="lang-auto">out1 = net1(x)

in2 = Variable(out1.data, requires_grad=True)
out2 = net2(in2)

loss = some_loss_function(out2)

loss.backward(retain_variables=True)
out1.backward(in2.grad.data)

# You can call loss.backward again here
# But not out1.backward !
</code></pre>

username: ajbrock
cooked: <p>Hmm, assuming detach() doesn’t free the graph, your way makes a lot of sense, and I think I could wrap that up into this chunk of code neatly. Thanks!</p>
<p>Edit: a slightly more elegant way to do this (still following your same idea) might be to copy out1 to in2 with the out-of-place detach:</p>
<p>in2 = out1.detach()</p>
<p>I’ll try out both and report back.</p>

username: alexis-jacq
cooked: <p>If in2 is detached, you can’t call loss.backward at all, since net2(in2) are not taken into account by the computation of the gradient.</p>
<p>When a variable is detached, the backward computations will not visit the branches that start from this variable (all the operations done on it).</p>
<p>So as <a class="mention" href="/u/alband">@albanD</a> suggested, you need to create a new graph (hence the new variable requiring a gradient) if you want to compute a gradient with respect to the new network (net2) independently on previous operation (net1).</p>

username: apaszke
cooked: <p>Both <a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/variable.py#L215-L220"><code>detach()</code> and <code>detach_()</code></a> will remove the reference to the creator, thus freeing the graph and buffers contained in it.</p>

username: 111319
cooked: <pre><code class="lang-python">import torch
x = torch.randn(10, 5, requires_grad = True)
w = x * x
a = w*w

a.sum().backward()
print(x.grad.sum())

w = x * x
a = w * w
w.detach_()

a.sum().backward()
print(x.grad.sum())
</code></pre>
<p>in this case, the output is:</p>
<pre><code class="lang-auto">tensor(27.7943)
tensor(55.5887)
</code></pre>
<p>Why the second backward can backpropagate the gradient to x? <code>detach_()</code> is an in-place operator and the <code>grad_fn</code> of <code>w</code> is changed to None, So from my understanding, <code>x</code> will not receive the second gradient. Do I get something wrong?</p>

username: albanD
cooked: <p>It is changing <code>w</code> inplace but it is not changing past use of it. So it still backprops fine because it was used before being detached.</p>

username: 111319
cooked: <p>Thanks for replay. But why sometimes inplace operator cause a error in backward and sometimes not?</p>

username: albanD
cooked: <p>Which error do you sometimes see and sometimes not?</p>

username: 111319
cooked: <p>Ou, I mean there are some inplace operators will cause an  error in backward, and some other operators will not. Back to the original question, is there a method that can detach a sub graph after a forward?</p>
<p>For example, there is a graph:<br>
<code>x -&gt; a -&gt; y -&gt; loss</code><br>
and I call the fast backward<br>
<code>loss -&gt; y -&gt; a -&gt; x</code>.<br>
After that I call<br>
<code>y.backward(grad=some-grad)</code><br>
and I only want to backward to a:<br>
<code>y -&gt; a</code>.</p>
<p>I know I can solve this by<br>
<code>x -&gt; x.detach() -&gt; a -&gt; y -&gt; loss</code><br>
and backward the gradient to “<code>a</code>” manually in the fast backward. But is there a better method?</p>

username: albanD
cooked: <aside class="quote no-group" data-username="111319" data-post="10" data-topic="1090">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111319/48/24495_2.png" class="avatar"> 111319:</div>
<blockquote>
<p>Back to the original question, is there a method that can detach a sub graph after a forward?</p>
</blockquote>
</aside>
<p>No there isn’t I’m afraid. We don’t allow modifying the graph.</p>
<p>If you just want the backward between two given Tensors, you should use the <code>grad_input = autograd.grad(output, input)</code> API. This will only run backward between the given Tensors.</p>