How to train a part of a network

alan_ayu: <p>I am building a model which consists of a subnet and some layers followed, the subnet has been trained before, and now I want to train the following new layers with the parameters of the subnet fixed.<br>
I want to know how to remove the parameters from the model.parameters() when optimizing the net using Adam.<br>
Any advice would be appreciated !!!</p>

colesbury: <p>The transfer learning tutorial has a relevant section:</p>
<p><a href="http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#convnet-as-fixed-feature-extractor" class="onebox" target="_blank">http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#convnet-as-fixed-feature-extractor</a></p>
<p>To summarize:</p>
<ol>
<li>
<p>Set <code>requires_grad=False</code> for all parameters you do not wish to optimizer. This avoids computing gradients for them:</p>
<pre><code class="lang-auto">for param in base_model.parameters():
 param.requires_grad = False
</code></pre>
</li>
<li>
<p>Call <code>.parameters()</code> on the part of the sub-network:</p>
<pre><code class="lang-auto">optim.Adam(model.sub_network.parameters(), ...)
</code></pre>
</li>
</ol>
<p>If your new layers aren’t entirely contained in a single Module, you can collect parameters by using list concatenation:</p>
<pre><code class="lang-auto">parameters = []
parameters.extend(model.new_layer1.parameters())
parameters.extend(model.new)layer2.parameters())
optimizer = optim.Adam(parameters, ...)
</code></pre>

alan_ayu: <p>Thank you very much, it works <img src="//discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:"></p>

roger_p: <p>Hi, maybe it’s silly, but if I have two sub_network, say netA and netB, the parameters of netA are put into optim.Adam (optim.Adam(model.netA.parameters())) while those of netB are not. Then what would happen to the netB’s parameters? Thanks!</p>

kaixin: <p>The parameters update when you call  <code>optimizer.step()</code>. Since your don’t have a optimizer for the parameters of netB, I think they won’t change.</p>

Rakshit_Kothari: <p>Is it necessary to set <code>.requires_grad = False</code>? If we simply not provide those layers to the optimizer, would it work? I understand we would free up memory by not computing gradient but is it necessary?</p>

ptrblck: <p>While <code>optimizer.step()</code> wouldn’t update these parameters (since you’ve never passed them to the optimizer), these parameters would still accumulate the gradients.<br>
This is of course wasteful, as these gradients are not needed (and Autograd could potentially stop the backward pass before reaching these parameters). Additionally to that, you would have to be careful, if you are planning to update these parameters in the future, e.g. by adding these parameters via <code>optimizer.add_param_group</code>, since they would already contain (large) gradients.</p>

qcyza: <p>Hi may I ask if I set some intermediate layer’s weight, require_grad = false, will the gradients still be able to backpropagate through the intermediate layers to the front layers so they get proper updates?</p>

ptrblck: <p>Yes, this will work as seen here:</p>
<pre><code class="lang-python"># setup
model = nn.Sequential(
    nn.Linear(1, 1),
    nn.Linear(1, 1),
    nn.Linear(1, 1)
)

# freeze middle layer
for param in model[1].parameters():
    param.requires_grad = False

# calculcate gradients
model(torch.randn(1, 1)).backward()

# check gradients
for name, param in model.named_parameters():
    print(name, param.grad)

&gt; 0.weight tensor([[-0.0005]])
  0.bias tensor([-0.0335])
  1.weight None
  1.bias None
  2.weight tensor([[-0.6302]])
  2.bias tensor([1.])
</code></pre>