Difference between state_dict and parameters()

Noam_Salomonski: <p>I saw two methods of accessing a model’s parameters:</p>
<ol>
<li>
<p><a href="https://stackoverflow.com/questions/49446785/how-can-i-update-the-parameters-of-a-neural-network-in-pytorch" rel="nofollow noopener">using state_dict</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/49201236/check-the-total-number-of-parameters-in-a-pytorch-model" rel="nofollow noopener">using parameters()</a></p>
</li>
</ol>
<p>Which is more correct?</p>
<p>What are the differences?</p>
<p>Thanks</p>

Kushaj: <p>model.parameters() contains all the learnable parameters of the model. state_dict() is a python dict mapping from layer to their parameters. Optimizers have their own state_dict. As state_dict are python dicts they are easy to save and load. <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict" rel="nofollow noopener">docs</a></p>

ptrblck: <p>Additionally to <a class="mention" href="/u/kushaj">@Kushaj</a>’s description, the <code>state_dict</code> holds all <code>buffers</code> besides the learnable parameters, e.g. <code>BatchNorm</code>'s running estimates, so that you can recreate your model properly.</p>

Kushaj: <p>I think should be added to docs as well.<br>
EDIT: It is in the docs my bad.</p>

ptrblck: <p>Well, in the docs you’ve posted this sentence might be misleading:</p>
<blockquote>
<p>Note that only layers with learnable parameters (convolutional layers, linear layers, etc.) have entries in the model’s <em>state_dict</em>.</p>
</blockquote>
<p>So I see your point here.<br>
Have a look at this simple model:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 3, 1, 1)
        self.bn1 = nn.BatchNorm2d(6)
        self.register_buffer('buf1', torch.randn(1))
        
    def forward(self, x):
        x = self.bn1(self.conv1(x))
        return x

model = MyModel()
print(model.state_dict()['buf1'])
</code></pre>
<p>As you can see <code>buf1</code> is also in the <code>state_dict</code>.<br>
Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> and propose a change?<br>
If not, let me know and I’ll do it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

Kushaj: <p>I will post the issue.</p>

Kushaj: <p>The issue is posted on Github. <a href="https://github.com/pytorch/pytorch/issues/17230" rel="nofollow noopener">link</a></p>

eplu: <p>Another difference appears to be that for all learnable parameters which appear in both <code>model.state_dict()</code> and <code>model.parameters()</code>,  <code>requires_grad</code> is <code>True</code> in the <code>model.parameters()</code> output and <code>False</code> in the <code>model.state_dict()</code> output.</p>
<p>Why is this?</p>

braindotai: <p>That’s true, not sure why exactly but after reloading my model using load_state_dict as -</p>
<pre><code class="lang-python">model.load_state_dict(torch.load(PATH))
</code></pre>
<p>It does everything as expected. Sets all the parameters respectively with <code>requires_grad</code> True.<br>
So my guess is in the state_dict the <code>requires_grad</code> is False in order to avoid any unnecessary autograd graph creation if the user does some computation to the tensors directly from state_dict.</p>

Noam_Salomonski: <p>I asked on Stack Overflow and got <a href="https://stackoverflow.com/questions/54746829/pytorch-whats-the-difference-between-state-dict-and-parameters" rel="noopener nofollow ugc">this answer</a>.</p>
<p>If something should be added or subtracted from it, please let me know.<br>
Otherwise, I will accept it here as well.</p>