Second order derivatives in meta-learning

username: hosseinshn
cooked: <p>I am implementing a meta-learning model that has meta-train and meta-test steps. For the meta-train, I optimize the parameters by</p>
<pre><code class="lang-auto">optimizer = optim.SGD(Model.parameters(),lr=1e-3, momentum=0.9, weight_decay = 5e-4)

grads_Model = torch.autograd.grad(meta_train_loss, Model.parameters(), ???#1)
fast_weights_Model = Model.state_dict()
adapted_params = OrderedDict()
for (key, val), grad in zip(Model.named_parameters(), grads_Model):
     adapted_params[key] = val - meta_step_size * grad
     fast_weights_Model[key] = adapted_params[key] 
</code></pre>
<p>For the meta-test phase, I calculate some other losses and add them together along with the meta-train loss to get a total loss and want to optimize it w.r.t the initial parameters:</p>
<pre><code class="lang-auto">meta_test_loss = loss1 + loss2 + loss3 (all obtained using adapted_state_dicts).  

total_loss = meta_train_loss + meta_test_loss
optimizer.zero_grad()
total_loss.backward(???#2)
optimizer.step()    
</code></pre>
<p>My question is how should I use create_graph=true and retain_graph=True for ???<span class="hashtag">#1</span> and ???<span class="hashtag">#2</span> positions to make it right. If I do create_graph=true for ???<span class="hashtag">#1</span> and leave ???<span class="hashtag">#2</span> blank I get an error that says I should retrain_graph=true for ???<span class="hashtag">#2</span>. If I do retain_graph=true for ???<span class="hashtag">#1</span> and leave ???<span class="hashtag">#2</span> blank it works, and if I do create_graph=true for ???<span class="hashtag">#2</span> is works with retain_graph=true for ???<span class="hashtag">#1</span>.<br>
I am confused and cannot figure out the correct form.<br>
To me, we are already making the adaptation on parameters in the meta-train and we just need to keep the graph so that we can backward through it in the meta_test step using the total loss and w.r.t to the initial parameters Model.parameters(). Am I correct or missing something here?</p>
<p>Thank you.</p>

username: dbp.pat94
cooked: <p>I have a similar doubt. What I have found so far is that nn.Parameters are LEAF nodes with no history. So, the Gradient Descent (GD) operations that you are trying to perform for the inner loop (adapted_params[key] = val - meta_step_size * grad) won’t be recorded in the computation graph.</p>
<p>The following may help you understand why updating model parameters like the way you have won’t work in PyTorch in more detail:</p>
<aside class="quote quote-modified" data-post="1" data-topic="78711">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jwillette/48/17366_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/cannot-calculate-second-order-gradients-even-though-create-graph-true/78711">Cannot calculate second order gradients even though `create_graph=True`</a> 
  </div>
  <blockquote>
    I have a training loop iteration that looks like this… 
mu, logvar = m(x)

alpha = torch.zeros(size, requires_grad=True)
loss = alpha * criterion(mu, logvar)

loss.backward(retain_graph=True, create_graph=True)
for p in m.parameters():
  p = p - LR * p.grad

x_next, y_next = data[j + 1]
mu_next, logvar_next = m(x_next)

meta_loss = criterion(mu_next, logvar_next)
grad_alpha = torch.autograd.grad(meta_loss, alpha)[0]

and pytorch is complaining that RuntimeError: One of the differentiated Tensors…
  </blockquote>
</aside>

<p>In fact, you may want to use the package “HIGHER” or refer to the following codes for a workaround (look for <strong>MetaModule</strong> in the repositories):</p>
<ul>
<li><a href="https://github.com/AdrienLE" class="inline-onebox" rel="noopener nofollow ugc">AdrienLE (Adrien Ecoffet) · GitHub</a></li>
<li><a href="https://github.com/xjtushujun/meta-weight-net/blob/bd1fd3e297c59df7c5264fb636c67d8ee03bcf0d/resnet.py#L240" rel="noopener nofollow ugc">https://github.com/xjtushujun/meta-weight-net/blob/bd1fd3e297c59df7c5264fb636c67d8ee03bcf0d/resnet.py#L240</a></li>
</ul>
<p>I hope that helps you understand the problem with your method.</p>

username: dbp.pat94
cooked: <p>If you go through the neat hack of using <code>MetaModule</code> instead of the <code>nn.Module</code> provided by PyTorch as mentioned in my first response to your question, you will be able to write and run your meta-learning algorithms.</p>
<p>However, it was not clear until recently why it works only that way. Keeping in mind that <code>nn.Parameters</code> don’t record history, please go through my most recent response here: <a href="https://discuss.pytorch.org/t/higher-order-derivatives-meta-learning/93051">https://discuss.pytorch.org/t/higher-order-derivatives-meta-learning/93051</a></p>
<p>Hopefully, this will help you understand how to find a work-around to implement meta-learning algorithms without having to use packages such as <a href="https://github.com/facebookresearch/higher" rel="nofollow noopener">higher</a>, <a href="https://github.com/tristandeleu/pytorch-meta" rel="nofollow noopener">pytorch-meta</a>, etc.</p>