How to save the requires_grad state of the weights

username: n0obcoder
cooked: <p>I define a model</p>
<blockquote>
<p>import torch<br>
from torchvision import models</p>
<p>model = models.resnet18()</p>
</blockquote>
<p>It has all its layers set as trainable (requires_grad = True for all layers)<br>
Then I freeze the final fc layer</p>
<blockquote>
<p>for param in model.fc.parameters():<br>
param.requires_grad = False</p>
</blockquote>
<p>Then I save the state_dict of the model</p>
<blockquote>
<p>torch.save(model.state_dict(), ‘model.pth’)</p>
</blockquote>
<p>Now I want to load the weights again. So I define the model once again and load the saved weights in it…</p>
<blockquote>
<p>model_new = models.resnet18()<br>
model_new.load_state_dict(torch.oad(‘model.pth’))</p>
</blockquote>
<p>Now when I print the requires_grad of its fc layer, all the requires_grad are back to the original setings (True)</p>
<blockquote>
<p>for param in model_new.fc.parameters():<br>
print(param.requires_grad)</p>
</blockquote>
<p>Its prints<br>
True<br>
True</p>
<p>So the question is that, how is the requires_grad setting getting changed on loading the weights?</p>
<p>Does saving the model.state_dict() even save the requires_grad settings?</p>

username: ptrblck
cooked: <p>The <code>state_dict</code> contains the tensors (basically the data), not the <code>nn.Parameters</code>, which hold the <code>requires_grad</code> attribute.<br>
Since you are recreating the model and load the <code>state_dict</code>, all flags are re-initialized.</p>
<pre><code class="lang-python">
model = models.resnet50()
print(model.fc.weight)
&gt; Parameter containing:
tensor([[-0.0046, -0.0034, -0.0171,  ..., -0.0081,  0.0019, -0.0050],
        [-0.0167, -0.0093,  0.0100,  ...,  0.0180,  0.0164,  0.0170],
        [ 0.0093, -0.0114,  0.0144,  ...,  0.0003,  0.0202,  0.0163],
        ...,
        [-0.0188, -0.0040, -0.0019,  ...,  0.0113, -0.0164, -0.0054],
        [ 0.0212, -0.0127,  0.0155,  ..., -0.0200,  0.0092,  0.0188],
        [-0.0179,  0.0083, -0.0003,  ..., -0.0012, -0.0048, -0.0127]],
       requires_grad=True)
sd = model.state_dict()
print(sd['fc.weight'])
&gt; tensor([[-0.0046, -0.0034, -0.0171,  ..., -0.0081,  0.0019, -0.0050],
        [-0.0167, -0.0093,  0.0100,  ...,  0.0180,  0.0164,  0.0170],
        [ 0.0093, -0.0114,  0.0144,  ...,  0.0003,  0.0202,  0.0163],
        ...,
        [-0.0188, -0.0040, -0.0019,  ...,  0.0113, -0.0164, -0.0054],
        [ 0.0212, -0.0127,  0.0155,  ..., -0.0200,  0.0092,  0.0188],
        [-0.0179,  0.0083, -0.0003,  ..., -0.0012, -0.0048, -0.0127]])
</code></pre>

username: n0obcoder
cooked: <p>Thanks for the clarification <a class="mention" href="/u/ptrblck">@ptrblck</a> ! : D</p>
<p>But what do you suggest me to do if i want to save the requires_grad flag for all the layers too ?</p>

username: John_Deterious
cooked: <p>I guess save the entire model.</p>

username: n0obcoder
cooked: <p>yes I tried</p>
<blockquote>
<p>torch.save(model, ‘path_name.pth’)</p>
</blockquote>
<p>and it seemed to work.</p>
<p>thanks <a class="mention" href="/u/john_deterious">@John_Deterious</a></p>
<p>and whats the best way to save the state of the optimizers ? why do we even need to save optimizer’s state? Can you pls explain this to me ?</p>
<p>Thanks in advance !!!</p>

username: n0obcoder
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="52906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>nn.Parameters</code> , which hold the <code>requires_grad</code> attribute</p>
</blockquote>
</aside>
<p>model = models.resnet50()</p>
<p>sd = model.state_dict()<br>
sd[‘fc.weight’] has both .data and .requires_grad attribute</p>
<p>So looking at the information that it holds it feels like saving the state_dict should be saving the requires_grad flag too…</p>
<p>Can you pls help me on that…</p>

username: ptrblck
cooked: <p>The <code>requires_grad</code> attribute you are printing from <code>sd['fc.weight'].requires_grad</code> is the default one for <code>torch.Tensor</code>s.</p>
<aside class="quote no-group" data-username="n0obcoder" data-post="6" data-topic="52906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/n0obcoder/48/16606_2.png" class="avatar"> n0obcoder:</div>
<blockquote>
<p>So looking at the information that it holds it feels like saving the state_dict should be saving the requires_grad flag too…</p>
</blockquote>
</aside>
<p>I don’t think so, as the <code>state_dict</code> is supposed to store only the parameters, not the computation etc.<br>
<a class="mention" href="/u/john_deterious">@John_Deterious</a>’s suggestion to store the complete model might work.</p>
<aside class="quote no-group" data-username="n0obcoder" data-post="5" data-topic="52906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/n0obcoder/48/16606_2.png" class="avatar"> n0obcoder:</div>
<blockquote>
<p>and whats the best way to save the state of the optimizers ? why do we even need to save optimizer’s state?</p>
</blockquote>
</aside>
<p>I would follow the <a href="https://github.com/pytorch/examples/blob/d587b53f3604b029764f8c864b6831d0ab269008/imagenet/main.py#L249">ImageNet example</a> and store a custom <code>dict</code> containing all data I would like to store. Some optimizers use internal buffers (e.g. running estimates etc.), that should also be stored/restored, if you plan on continuing the training.</p>