For freezing certain layers, why do I need a two step process?

paganpasta: <p>I see almost all responses (<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#convnet-as-fixed-feature-extractor" rel="noopener nofollow ugc">tutorial</a>, <a href="https://discuss.pytorch.org/t/how-to-train-a-part-of-a-network/8923/2">discussion</a>) on training part of a network to include these 2 steps</p>
<ol>
<li>Set <code>target</code> network parameters to <strong>requires_grad=False</strong>
</li>
<li>Pass only <code>non-target</code> parameters to the optimiser</li>
</ol>
<p>Doing any one of the above two can achieve the effect of not updating the <code>target</code> layers</p>
<p>My take on the comparison is as follows.</p>
<ol>
<li>Setting <code>requires_grad=False</code> requires lower compute as <code>target</code> gradients are not computed.</li>
<li>Excluding from the optimiser makes the <code>optim.step()</code> faster as it does not have  to loop over all param groups.</li>
<li>Setting <code>requires_grad=True</code> and skipping the layers in the optimiser still leads to gradient computation. Maybe an eventual gradient overflow due to accumulation?</li>
<li>Setting <code>requires_grad=False</code> and still passing to optimiser has <strong>some</strong> impact of <code>optimiser.step()</code> call</li>
</ol>
<p>Am I missing anything?</p>

ptrblck: <ol>
<li>Yes and this can be verified using this small code snippet:</li>
</ol>
<pre><code class="lang-python">import torch
import torch.nn as nn


model = nn.Sequential(
        nn.Conv2d(3, 3, 3),
        nn.Conv2d(3, 3, 3),
        nn.Conv2d(3, 3, 3)
).cuda()

#model[1].weight.requires_grad = False
#model[1].bias.requires_brad = False

x = torch.randn(1, 3, 24, 24).cuda()
out = model(x)
out.mean().backward()
</code></pre>
<p>In the use case where all layers are trainable you would see 2 <code>dgrad</code> and 3 <code>wgrad</code> kernels:</p>
<pre><code class="lang-auto"> Time (%)  Total Time (ns)  Instances  Avg (ns)  Med (ns)  Min (ns)  Max (ns)  StdDev (ns)                                                  Name                                                
 --------  ---------------  ---------  --------  --------  --------  --------  -----------  ----------------------------------------------------------------------------------------------------
     21.2            16544          2    8272.0    8272.0      8256      8288         22.6  void cudnn::cnn::dgrad2d_grouped_direct_kernel&lt;float, int, float, float, (bool)0, (bool)1, (int)0, …
     20.9            16288          3    5429.3    4896.0      4768      6624       1036.6  void cudnn::cnn::wgrad2d_grouped_direct_kernel&lt;(bool)0, (bool)1, int, float, float, float&gt;(cudnn::c…
     19.8            15456          3    5152.0    4928.0      4768      5760        532.6  void implicit_convolve_sgemm&lt;float, float, (int)128, (int)5, (int)5, (int)3, (int)3, (int)3, (int)1…
</code></pre>
<p>If you freeze the middle layer, the one <code>wgrad</code> kernel will be missing as the weights do not get any gradient. The <code>dgrad</code> kernel is still called as the gradients needs to be backpropagated to the first conv layer:</p>
<pre><code class="lang-auto"> Time (%)  Total Time (ns)  Instances  Avg (ns)  Med (ns)  Min (ns)  Max (ns)  StdDev (ns)                                                  Name                                                
 --------  ---------------  ---------  --------  --------  --------  --------  -----------  ----------------------------------------------------------------------------------------------------
     22.2            15968          2    7984.0    7984.0      7968      8000         22.6  void cudnn::cnn::dgrad2d_grouped_direct_kernel&lt;float, int, float, float, (bool)0, (bool)1, (int)0, …
     21.7            15552          3    5184.0    4896.0      4768      5888        613.0  void implicit_convolve_sgemm&lt;float, float, (int)128, (int)5, (int)5, (int)3, (int)3, (int)3, (int)1…
     15.2            10944          2    5472.0    5472.0      4736      6208       1040.9  void cudnn::cnn::wgrad2d_grouped_direct_kernel&lt;(bool)0, (bool)1, int, float, float, float&gt;(cudnn::c…
</code></pre>
<ol start="2">
<li>
<p>Yes and is also more “explicit” assuming you never want to train these frozen parameters.</p>
</li>
<li>
<p>Yes. Not only will the gradients be computed but will also be accumulated to the <code>.grad</code> attribute, which will launch another kernel.</p>
</li>
<li>
<p>This depends on the actual use case and e.g. if this (now) frozen parameter was updated before, if the optimizer uses internal running stats, thus if these stats were already populated for the frozen parameter, etc. In such a case even frozen parameters could see an update if you are setting the gradients to zero (i.e. are not using <code>zero_grad(set_to_none=True)</code> which is the default in newer PyTorch releases) and if the optimizer can use its internal state to update this parameter.</p>
</li>
</ol>