Passing to the optimizers frozen parameters

username: Valerio_Biscione
cooked: <p>When I want to do transfer learning, I set the require_grad = False and only pass the other parameters to the optimizer. But what happens if I only do one of these two steps?</p>
<ol>
<li>What happens if I set several modules’ reguire_grad = False but then I pass all net.parameters() to the optimize.</li>
<li>What happens if I keep the require_grad = True but only pass a subset of the parameters to the optimizer (the subset I want to fine tune, for example).</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>The parameters with <code>requires_grad = False</code> won’t accumulate a gradient during the backward pass and their <code>.grad</code> attribute will stay <code>None</code> by default (if no previous gradient was calculated).<br>
The optimizer will skip all parameters with a <code>None</code> gradient as seen <a href="https://github.com/pytorch/pytorch/blob/b7b99ab0c8f82100177729b9751481852d83e77e/torch/optim/sgd.py#L95-L96">here</a>.</p>
</li>
<li>
<p>All parameters will accumulate gradients and the optimizer will only update the passed parameters. If you call <code>optimizer.zero_grad()</code> and don’t use <code>model.zero_grad()</code>, the “unused” parameters will continue to accumulate gradients. If you later decide to use these parameters in another optimizer, you should make sure to zero out their gradients before.</p>
</li>
</ol>

username: Valerio_Biscione
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
does it mean that (1) and <em>“setting require_grad = False and only passing the other parameters to the optimizer”</em> are equivalent?</p>

username: ptrblck
cooked: <p>They will yield the equivalent results, yes.</p>
<p>Filtering out the parameters is explicit and could thus increase the code readability and will also avoid iterating over parameters without a <code>grad</code> attribute in the <code>step</code> method.<br>
Approach (1) would allow you to unfreeze the parameters later in training directly and train these parameters without adding these via <code>add_param_group</code>.<br>
I guess it might depend a bit on your coding style and what your exact use case is (e.g. are you planning on unfreezing or not?).</p>

username: Valerio_Biscione
cooked: <p>thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>, as usual very informative and precise answers . Maybe some of this info should be added to this tutorial <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a></p>

username: ptrblck
cooked: <p>This might make sense. Would you be interested in creating an issue to discuss the current shortcomings and create a fix for it after the discussion with the code owners? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Giuliano_Albanese
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/valerio_biscione">@Valerio_Biscione</a> did you happen to create an issue to integrate this in the docs? I think this is still not mentioned in the finetuning tutorial mentioned above</p>

username: ptrblck
cooked: <p>No, I did not. Would you be interested in adding information that you think is missing to the docs?</p>

username: Valerio_Biscione
cooked: <p>no, I haven’t, sorry</p>