Way to freeze layers

sneer: <p>Hi,<br>
I want to freeze (some) layers of a network feature encoder (resnet50 in my case) and then add some dense layer to the feature encoder to evaluate on some classification task.</p>
<p>I am freezing the layers like this:</p>
<pre><code class="lang-python">for child in self.feature_extractor.children():
    for param in child.parameters():
        param.requires_grad = False
</code></pre>
<p>First questions: Is this the correct way? I saw some people use <code>named_children()</code> instead of <code>children()</code>. Also some people call <code>parameters</code> in the first for loop and leave out the second one. Are there any difference between those methods?</p>
<p>I also saw some people to only pass layers that have <code>requires_grad=True</code> to the optimizer.<br>
<code>optimizer = optim.SGD(filter(lambda p: p.requires_grad, self.parameters()), lr=0.1)</code><br>
Second question: Is this required after using the above method?</p>
<p>To add the dense layer I make a new layer with <code>self.classifier=nn.Linear(...)</code> and use it in the <code>forward()</code> function like:</p>
<pre><code class="lang-python">representations = self.feature_extractor(x).flatten(1)
return self.classifier(representations)
</code></pre>
<p>using the optimizer like this (passing <code>self.parameters()</code> as params argument:<br>
<code>optimizer = optim.SGD(params=self.parameters()), lr=0.1)</code><br>
should work fine, right? And should calculate gradients for the  classifier (and the feature extractor in case I don’t want to freeze it)</p>
<p>Thanks a lot</p>

ptrblck: <aside class="quote no-group" data-username="sneer" data-post="1" data-topic="158016">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b9e5f3/48.png" class="avatar"> sneer:</div>
<blockquote>
<p>I saw some people use <code>named_children()</code> instead of <code>children()</code></p>
</blockquote>
</aside>
<p>The latter will return the <code>name</code> as well as the child module which could be useful to e.g. filter out child modules if needed.</p>
<aside class="quote no-group" data-username="sneer" data-post="1" data-topic="158016">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b9e5f3/48.png" class="avatar"> sneer:</div>
<blockquote>
<p>Also some people call <code>parameters</code> in the first for loop and leave out the second one. Are there any difference between those methods?</p>
</blockquote>
</aside>
<p><code>.parameters()</code> will return all parameters of the module and all internally registered submodules (children). Your approach would work, but you could also directly use <code>self.feature.extractor.parameters()</code>.</p>
<aside class="quote no-group" data-username="sneer" data-post="1" data-topic="158016">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b9e5f3/48.png" class="avatar"> sneer:</div>
<blockquote>
<p>Second question: Is this required after using the above method?</p>
</blockquote>
</aside>
<p>It’s not required, as the optimizer will ignore parameters, which do not require gradients. Depending on your use case you might want to set the <code>requires_grad</code> attribute to <code>True</code> after a while at which point the optimizer would then start optimizing these parameters. If that’s not your use case, you could just filter out the parameters from the beginning to make sure they are never optimized.</p>

sneer: <p>Perfect,<br>
thanks you very much</p>