Why save optimizer state dict?

username: snowe
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Probably a simple question but I can’t seem to figure out why exactly we should save the <code>optimizer.state_dict()</code> to resume training? Why isn’t it sufficient to save the <code>model.state_dict()</code>? The  <code>optimizer.state_dict()</code> does not have any learnable parameters if I understand it correctly…</p>
<p>Thank you for the clarification!</p>
<p>All the best,<br>
snowe</p>

username: googlebot
cooked: <p>most optimizers collect per-parameter gradient statistics, so you’d save things like gradient’s running mean &amp; variance, and resumed training would make better initial steps.</p>

username: snowe
cooked: <p>Hi <a class="mention" href="/u/googlebot">@googlebot</a>, thank you for your response! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>But theoretically it would also be possible to resume training without the optimizer for a loss in initial performance?</p>
<p>If for example I want to take the pretrained parameters and use them in another network with a different structure. I can do that for the <code>model.state_dict()</code> with the <code>strict=False</code> flag, but there is no such thing for the <code>optimizer.state_dict()</code>…</p>

username: googlebot
cooked: <p>There is a difference between using pretrained parameters and resuming training, if you “transfer” parameters, gradient history no longer applies and so optimizer should start from a clean state.</p>

username: snowe
cooked: <p>I don’t exactly understand what you mean…</p>
<p>From what I understand, if you want to use a pretrained model to test it (no more learning), you can use the <code>model.state_dict()</code> to load the ‘best’ weights. If you want to continue training you need both, the <code>model.state_dict()</code> and the <code>optimizer.state_dict()</code>.</p>
<p>My problem applies to the second use case, where I want to continue training but my model is slightly different. So I am not sure whether I still need the <code>optimizer.state_dict()</code> and if so, how I can apply it to a new model.</p>

username: googlebot
cooked: <p>No, you’d reload optimizer’s state_dict if you want to pause/resume training at epoch N&gt;0 for whatever reason. If model or dataset changes, that should be considered a new run from epoch 0; you’re free to reload parameters from model.load_state_dict(strict=False) for it, there is no need for old optimizer’s state (it only contains stale auxiliary buffers).</p>
<p>PS even if you’re training reloaded params further, new run will have differ gradient statistics, so optimizer’s reload is not good</p>

username: snowe
cooked: <p>Ah I see, that makes sense. Thank you!</p>
<p>But loading the pretrained parameters will still speed up the training right, even if the network and the data are not exactly the same? They should be considerably better than random weights?</p>

username: googlebot
cooked: <p>Sure, you’re just resetting learning rate “adaptations” from old training.</p>