Performing backward on another backward

username: Jason_Lin
cooked: <p>Hi folks! I ran into the following issue. I was wondering if someone could help me out.</p>
<p>I have a loss <code>l(w,theta)</code>, where <code>w</code> is weights and <code>theta</code> is network parameters. I need to first calculate <code>theta.hat</code> using S.G.D. as follows.</p>
<pre><code class="lang-auto">theta.hat = theta - alpha * grad l(w,theta).
</code></pre>
<p>Then, I need to use the updated <code>theta.hat</code> in the same network and calculate another loss <code>L(theta.hat)</code>. As <code>theta.hat</code> is a function of <code>w</code>, so is <code>L</code>. My question is, how can I get the gradient of <code>L</code> with respect to <code>w</code>?</p>
<p>All my attempts had the following error: One of the differentiated Tensors appears to not have been used in the graph. So I guess <code>w</code> was not really in my graph.</p>
<p>For example, I tried using <code>optimizer.step()</code>, but after that, <code>theta.hat</code> got the updated values, but <code>w</code> was lost from the graph. In other words, <code>optimizer.step()</code> was sort of like <code>with torch.no_grad()</code>, whereas I would like something that kept the graph that traced back to <code>w</code>.</p>
<p>Thanks for your help!</p>

username: albanD
cooked: <p>Hi,</p>
<p>To do this, you will need to set <code>create_graph=True</code> in the first call to backward !</p>

username: Jason_Lin
cooked: <p>Thanks for your suggestion. I tried and the same thing happened. Could you take a look at the following example? The goal is to get the gradient of <code>l_val</code> with respect to <code>eps</code>.</p>
<pre><code class="lang-auto">def cal_loss(net, x_train, y_train, x_val, y_val):
    
    y_train_hat = net(x_train)
    cost = nn.CrossEntropyLoss(reduction='none')(y_train_hat, y_train)

    # here is eps, not sure whether to set it to nn.Parameters, but either way didn't work
    eps = nn.Parameter(torch.zeros_like(cost), requires_grad=True)
    l_train = torch.sum(cost * eps)
    
    # first backward step
    opt = optim.SGD(net.parameters())
    opt.zero_grad()
    l_train.backward(create_graph=True)
    opt.step()
    
    # use updated network to calculate l_val
    y_val_hat = net(x_val)
    l_val = nn.CrossEntropyLoss()(y_val_hat, y_val)

    # this didn't work
    grad_eps = torch.autograd.grad(l_val, eps)[0]
</code></pre>
<p>The last line got an error: One of the differentiated Tensors appears to not have been used in the graph. I suspected <code>opt.step()</code> was the culprit, but I’m not sure exactly which line left <code>eps</code> out of the graph. Any ideas how to fix this? Thanks so much!</p>
<p>Jason</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes one issue is that you are trying to differentiate through the optimizer step. But pytorch optimizers are not differentiable.<br>
You can take a look at libraries like <a href="https://github.com/facebookresearch/higher/">higher</a> on how to go around this.</p>

username: Jason_Lin
cooked: <p>Thanks I’ll look into that <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>