Gradient with respect to parameters that update model parameters?

danthekang: <p>I am trying to implement this <a href="https://arxiv.org/pdf/1803.09050.pdf" rel="noopener nofollow ugc">paper</a> for a generic network. I have found an <a href="https://github.com/danieltan07/learning-to-reweight-examples" rel="noopener nofollow ugc">implementation </a> that works, but it requires a definition of the networks in other custom designed class module (MetaModule), which is not too scalable.</p>
<p>The algorithm that I am trying to implement is the following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b5e1138e185cfac804412ff7a4fc0573e91bf431.png" alt="image" data-base62-sha1="pWYE3ysDYNJjme6l4RJkcZ3Dp1T" width="401" height="402"></p>
<p>And the problem I am encountering is the following:<br>
Code:</p>
<pre><code class="lang-auto">#meta_net is created using the same network structure of a  model and the params are copied over

# Line 4
y_train_meta_hat = meta_net(x_train)

# Line 5: Compute loss function using our prediction
cost = loss_fn(y_train_meta_hat, y_train, reduction='none')
eps = Variable(torch.zeros(cost.size()), requires_grad=True)

# Line 6: get gradient with respect to parameters
l_f = torch.sum(cost * eps)
meta_net.zero_grad()
grads = torch.autograd.grad(l_f, (meta_net.parameters()), retain_graph=True, create_graph=True)

# Line 7: update theta values of the network
for tgt, src in zip(meta_net.named_parameters(), grads):
    tgt.param.data -= learning_rate * src
 
# Line 8: compute the value of validation data using updated theta values
y_val_hat = meta_net(x_val)

# line 9-10: compute loss function, and compute gradient with respect to eps
l_g = loss_fn(y_val_hat, y_val, reduction='mean')
grad_eps = grad(l_g, eps)   ### this does not work
</code></pre>
<p>Specifically, I am having trouble with the last line, taking gradient of l_g with respect to eps. The gradient computation graph seems detached the moment I update the parameters using first gradient ( tgt.param.data -= learning_rate * src). To be more specific, I get the following error:<br>
<em>“One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.”</em></p>
<p>Could anyone give some guidance on how to resolve this issue?</p>

yhqjohn: <p>Parameters are designed to be leaf nodes in PyTorch. You should not backpropagate parameters. And you are right, modifying .data of a tensor does not produce a grad_fn.</p>
<p>You can use customized layers with backpropagatable “parameters”. An example is <a href="https://github.com/dragen1860/MAML-Pytorch" rel="nofollow noopener">https://github.com/dragen1860/MAML-Pytorch</a></p>
<p>Personally， I suggest you to use my lib <a href="https://github.com/yhqjohn/MetaNN%EF%BC%8C" rel="nofollow noopener">https://github.com/yhqjohn/MetaNN，</a> and use dependents() instead of parameters().</p>

Zeeky: <p>Did you find a way to bypass this problem?</p>
<p>It’s just computing for the gradients after the parameter updates (line 6 and line10). I’m shocked that pytorch does not have a way to implement this algorithm without re-writing the whole nn.Module.</p>

David_Blom: <p>Have you found a solution to this problem?</p>