How do I calculate the gradients of a non-leaf variable w.r.t to a loss function?

Abhai_Kollara: <p>If my understanding is correct, calling the <code>.backward()</code> of a <code>Variable</code> only generates the gradients of the leaf nodes. Is there any way to calculate the gradients w.r.t to an arbitrary Variable, sort of like <code>torch.gradients(loss, variables)</code></p>

tom: <p>Hello <a class="mention" href="/u/abhai_kollara">@Abhai_Kollara</a></p>
<p>Update:<br>
The proper solution is to use <code>.retain_grad()</code></p>
<pre><code class="lang-python">v = torch.autograd.Variable(torch.randn(3), requires_grad=True)
v2 = v+1
v2.retain_grad()
v2.sum().backward()
v2.grad
</code></pre>
<p>Apparently, this is common enough.</p>
<p>This is what I had posted before I knew better:</p>
<p>how about using hooks, e.g.</p>
<pre><code class="lang-python">v = torch.autograd.Variable(torch.randn(3), requires_grad=True)
def require_nonleaf_grad(v):
     def hook(g):
         v.grad_nonleaf = g
     v.register_hook(hook)
v2 = v+1
require_nonleaf_grad(v2)
v2.sum().backward()
v2.grad_nonleaf
</code></pre>
<p>I don’t recommend calling it <code>.grad</code> to not collide with pytorch internals.</p>
<p>Best regards</p>
<p>Thomas</p>

amirid: <p>Hello <a class="mention" href="/u/tom">@tom</a></p>
<p>I want to get the gradients of the  discriminator network w.r.t the fake variable generated by the generator net. I tried both of your solutions, by using the first way I get None for the gradient value and by the second solution I get the following error:<br>
AttributeError: ‘Tensor’ object has no attribute ‘grad_nonleaf’.</p>
<p>Do you have any idea of what’s wrong?</p>

tom: <p>Most likely, your generator output doesn’t require gradients, maybe put it in eval or set requires_grad=False on its parameters.</p>
<p>Best regards</p>
<p>Thomas</p>

JoeHEZHAO: <p>Dear Thomas</p>
<p>Thanks for your posting. I am also encountering the ‘<strong>optimize non-leaf variable</strong>’ issue and I am grategul if you can provide some feedback to my practice.</p>
<p>What I want to do is using a linear combination (<strong>learnable</strong>) of  basis filters for convolution operation and optimize <strong>only the linear coefficients</strong> as below:</p>
<pre><code class="lang-auto">class net(torch.nn.Module):
    def __init__():
        fitler_basis = [f_height, f_weight, num_filter];
        self.coeff = [num_filter, c_in, c_out]; # To be optimized

        self.conv2d = torch.nn.conv2d(c_in, c_out, strides=1, padding=1)
        self.conv2d.weight = torch.matmul(filter_basis, coeff) # linear combinaton of pre-fixed filter as weights;

    def forward(self, data):
        return self.conv2d(data)

    def train(self):
        loss = ...
        optim(self.coeff, loss)
</code></pre>
<p>How should I achieve my goal ?</p>

Deep_Patel: <p>Perfect solution for getting gradients of non-leaf variables in autograd</p>

TriKri: <p>Hi <a class="mention" href="/u/tom">@tom</a>,</p>
<p>Thanks for the <code>require_nonleaf_grad(my_tensor)</code> solution!</p>
<p>However, when I use it, the memory on my GPU quickly fills up, and even if I delete <code>my_tensor.grad_nonleaf</code> imediatelly after having called <code>loss.backward</code>, the gradient tensor is still preserved in exactly 50% of the cases. I know this because I store <code>my_tensor.storage().data_ptr()</code> in the set <code>addresses</code>, and each 500th training batch I use Python’s built-in garbage collector to find existing tensors and count the number of tensors <code>t</code> for which <code>t.storage().data_ptr() in addresses</code>, and this number increases with 250 each time I do so.</p>
<p>Do you know how that can be? I don’t even use <code>my_tensor.grad_nonleaf</code> for anything yet, I just request that it be created and then delete is as soon as I get the opportunity. How can I debug this most easily?</p>

tom: <p>Note that the proper solution is to use retain_grad.<br>
It’s really easy to create circular dependencies with gradients and Tensors, so maybe you can set the gradient to None explicitly.<br>
Other than that, that it happens every second time might point to something else being up with your code, too. It’s hard to tell.</p>
<p>Best regards</p>
<p>Thomas</p>

TriKri: <p>Thanks for your answer! I changed from using <code>require_nonleaf_grad</code> to using <code>retain_grad</code> and now the problem seems to be gone.</p>