Passing Params to an Optimizer

username: Epoching
cooked: <p>Hey!</p>
<p>A quick question regarding the code snippet below from the PyTorch <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="nofollow noopener">Autograd Mechanics Tutorial</a>:</p>
<pre><code class="lang-auto">model = torchvision.models.resnet18(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
# Replace the last fully-connected layer
# Parameters of newly constructed modules have requires_grad=True by default
model.fc = nn.Linear(512, 100)

# Optimize only the classifier
optimizer = optim.SGD(model.fc.parameters(), lr=1e-2, momentum=0.9)
</code></pre>
<p>What would be the difference if I were to just pass in the entire model’s parameters? E.g.:</p>
<pre><code class="lang-auto">optimizer = optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">optimizer = optim.SGD(model.fc.parameters(), lr=1e-2, momentum=0.9)
</code></pre>
<p>I am a bit confused why this would matter, since we are setting <code>requires_grad=False</code> to every parameter of the model, besides the <code>model.fc</code> layer (because this layer is created as a new module, which has <code>requires_grad=True</code> by default.</p>
<p>Thanks and I apologize if this has been asked already,<br>
Epoching</p>

username: ptrblck
cooked: <p>In this example it wouldn’t change the behavior, and the optimizer will just skip all parameters without a gradient.<br>
However, “explicit is better than implicit” according to the Python Zen, so the the first approach might be cleaner.</p>
<p>There are some edge cases, where e.g. a parameter was trained, then frozen and should therefore not be updated anymore.<br>
However, if you are using weight decay, this "frozen’ parameter might still be updated.<br>
As you can see, this is not the usual use case, but might create some headache while debugging. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Epoching
cooked: <p>Wow very interesting!</p>
<p>Just to confirm, this edge case you are talking about would be in favor of the 2nd option right? (i.e. use all model parameters so you have the ability to freeze them during training, but risky due to weight decay)</p>
<p>Thanks for the reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Epoching" data-post="3" data-topic="63191">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/epoching/48/18395_2.png" class="avatar"> Epoching:</div>
<blockquote>
<p>Just to confirm, this edge case you are talking about would be in favor of the 2nd option right?</p>
</blockquote>
</aside>
<p>Yes, that’s correct. However, this edge case was an “error” in some posts and rather unwanted behavior. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: Epoching
cooked: <p>I see, thanks for the clarification <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>