What is the typical implementation when you train an additional optimizer for an additional layer?

JimW: <p>Hi, I had a network called model() and it generated predictions like this:</p>
<pre><code class="lang-auto">pred1= model(input1)
pred2= model(input2)

all_preds = torch.concat([pred1, pred2]).flatten()
fc_new = nn.linear(256, 10)
opt_new = torch.optim.Adam(fc_new.parameters(), lr=0.1)
loss_new = my_loss_func(all_preds, labels)
loss_new.backward()
opt_new.step()
</code></pre>
<p>But I got the following error:<br>
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
<p>Basically,  what I want to do is,  freeze “model” but train “fc_new”. But “fc_new” needs to consume the output of “model” as input.</p>
<p>Can someone provide me some instructions on how to fix this problem?  Thanks.</p>

J_Johnson: <p>Try this:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(list(model.parameters())+list(fc_new.parameters()), lr = 0.1)
</code></pre>

KFrank: <p>Hi Jim!</p>
<aside class="quote no-group quote-modified" data-username="JimW" data-post="1" data-topic="170800" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jimw/48/38000_2.png" class="avatar"> JimW:</div>
<blockquote>
<pre><code class="lang-auto">pred1= model(input1)
pred2= model(input2)

all_preds = torch.concat([pred1, pred2]).flatten()
fc_new = nn.linear(256, 10)
opt_new = torch.optim.Adam(fc_new.parameters(), lr=0.1)
loss_new = my_loss_func(all_preds, labels)
loss_new.backward()
opt_new.step()
</code></pre>
<p>Basically,  what I want to do is,  freeze “model” but train “fc_new”. But “fc_new” needs to consume the output of “model” as input.</p>
</blockquote>
</aside>
<p>Note, in the code you posted, you never actually <em>use</em> <code>fc_new</code> – you don’t<br>
have <code>fc_new</code> consuming <code>all_preds</code>.</p>
<p>Based on what I think you want, you could do this:</p>
<pre data-code-wrap="python"><code class="lang-python">fc_new = nn.linear(256, 10)
opt_new = torch.optim.Adam(fc_new.parameters(), lr=0.1)

with torch.no_grad:   # won't construct computation graph and pred1 and pred2 won't carry requires_grad = True
    pred1= model(input1)
    pred2= model(input2)

all_preds = torch.concat([pred1, pred2]).flatten()

new_preds = fc_new (all_preds)   # which I think you want

loss_new = my_loss_func (new_preds, labels)
loss_new.backward()
opt_new.step()
</code></pre>
<p>This approach to “freezing” <code>model</code> is possible because <code>model</code> comes before<br>
<code>fc_new</code> – that is, you are not feeding the output of <code>fc_new</code> into the part you<br>
want frozen – and because you have easy access to <code>model</code> and <code>fc_new</code><br>
separately (rather than them both being part of some larger model that you<br>
would have to dig down into).</p>
<p>Best.</p>
<p>K. Frank</p>