Loading optimizers state_dict throws an error

n0obcoder: <p>I am not sure how model and optimizers work together in pytorch.</p>
<p>Here is the thing,</p>
<p>I save my model(full model, not the state_dict of the model) and i save the optimizer.state_dict</p>
<pre><code class="lang-auto">torch.save({
            'model': model, # it saves the whole model
            'optimizer_state_dict': optimizer.state_dict(),
            'lr_scheduler_state_dict': lr_scheduler.state_dict(),
            }, save_path)
</code></pre>
<p>Then i load my model and freeze some layers and i define the optimizer again and then load the optimizers state_dict…</p>
<pre><code class="lang-auto">ckpt = torch.load(save_path) 
model = ckpt['model']

for name, param in model.named_parameters():
    if ('layer4' in name) or ('fc' in name):
        param.requires_grad = True 
    else:
        param.requires_grad = False
        
optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr = lr)
optimizer.load_state_dict(ckpt['optimizer_state_dict'])

exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)
exp_lr_scheduler.load_state_dict(ckpt['lr_scheduler'])
</code></pre>
<p>It throws an ugly error-<br>
<strong>ValueError: loaded state dict contains a parameter group that doesn’t match the size of optimizer’s group</strong></p>
<p>Can you pls help me get it corrected ?</p>
<p>Any why do we even need to save the state_dict of the optimizers and the scheduler ?</p>

liyz15: <p>It could be you filter out some parameters in the new optimizer, resulting mismatch. Removing the filter should resolve the issue.</p>
<p>Saving the state_dict of optimizer and scheduler helps to resume training.</p>

n0obcoder: <p><a class="mention" href="/u/liyz15">@liyz15</a>, thanks for replying <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I get it now, that the error is because of the mismatch of the parameters<br>
but i  can not remove the filters, since i am freezing some layers of the model.</p>
<p>What if after freezing the layers, i define a new optimizer ?<br>
Would it be any different from loading the state_dict of the previously saved optimizer ?</p>

liyz15: <p>I believe setting <code>requires_grad=False</code> should be enough to freeze. See <a href="https://stackoverflow.com/questions/53159427/pytorch-freeze-weights-and-update-param-groups" rel="nofollow noopener">https://stackoverflow.com/questions/53159427/pytorch-freeze-weights-and-update-param-groups</a></p>
<p>As long as you set <code>requires_grad=False</code> before forward and use <code>optimizer.zero_grad()</code>, it won’t be updated.</p>
<p>Adam optimizer maintains learning rate adaptively, so it has internal state changing with training. It’s different between new and trained. Generally, if you want to continue training, load from state_dict.</p>

n0obcoder: <p>Hi <a class="mention" href="/u/liyz15">@liyz15</a><br>
Thanks for the explaination once again ! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
But what do you  suggest me to do in my case where, i have to freeze layers after some epochs of training ?<br>
Do you suggest me to define a new optimizer after freezing the layers of the model ?</p>

liyz15: <p>What’s the purpose of freezing layers? If you are trying to finetune on different dataset, a new one is preferred. If it’s some training techniques to freeze some layer during training, then continue with the same one.</p>
<p>BTW, saving directly the entire model may not be the best practice, save <code>model.state_dict()</code> is recommended, see <a href="https://github.com/pytorch/pytorch/blob/761d6799beb3afa03657a71776412a2171ee7533/docs/source/notes/serialization.rst" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/761d6799beb3afa03657a71776412a2171ee7533/docs/source/notes/serialization.rst</a></p>

n0obcoder: <p>Yes freezing layers is a technique of training a model.  So i train the model for, lets say 5 epochs on the last 3 layers, then train further for 3 epochs for only the last 2 layers(freezing the thirds last layer) and so on…</p>
<p>And saving the model.state_dict() does not save the requires_grad attribute of the parameters of the model whearas saving the entire model does save it. Saving the entire model works as long as u are not changing the architecture of the model itself.</p>
<p>So, given that i have to freeze layers after a few epochs, do i have any option other than defining a new optimizer ?</p>

liyz15: <p>Just use the old one, no need for filtering, <code>requires_grad=False</code> will do freezing.</p>