ValueError: loaded state dict has a different number of parameter groups

Liana_Baghdasaryan1: <p>Hi,</p>
<p>I trained a model and than continued to train the same model with an additional layer, where I needed to add parameters to the optimizer with this line</p>
<p>scheduled_optim._optimizer.add_param_group({‘params’: model.speaker_encoder.stl.parameters()})</p>
<p>Note here that the speaker_encoder.stl is the new added layer.</p>
<p>than I  added a new loss to the existing model and wanted to continue training this model with loaded optimizer. But it throws an error<br>
ValueError: loaded state dict has a different number of parameter groups</p>
<p>What can you suggest?</p>

ptrblck: <p>I don’t know which code raises the error exactly, as it seems you might be trying to load an optimizer <code>state_dict</code> after manipulating the parameters?<br>
If so, could you try to restore the <code>optimizer</code> before adding new parameter groups?</p>

Liana_Baghdasaryan1: <p>Lets imagine a 3 stage training.</p>
<ol>
<li>I am training some model and save at 100k step to continue.</li>
<li>I need to add additional parameters to the previous model and train more 100k steps.<br>
For that I am adding parameters of the model after loading the model and optimizer. Then, I add model.speaker_encoder.stl this layer to the model and scheduled_optim._optimizer.add_param_group({‘params’: model.speaker_encoder.stl.parameters()})<br>
these params to the optimizer. It worked fine. I saved the model and optimizer at step 200k.</li>
<li>Now I want to load model at 200k step and use additional loss for the training. When I am doing this it says.<br>
ValueError: loaded state dict has a different number of parameter groups</li>
</ol>

Liana_Baghdasaryan1: <p>I reproduced it with a small example. Imagine I have a network which  I train a few epochs and save the state of model and optimizer.</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 6, 5)<br>
self.pool = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(6, 16, 5)<br>
self.fc1 = nn.Linear(16 * 5 * 5, 120)<br>
self.fc2 = nn.Linear(120, 84)<br>
self.fc3 = nn.Linear(84, 10)<br>
<span class="hashtag">#self</span>.fc4 = nn.Linear(10, 10)</p>
<pre><code>def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = torch.flatten(x, 1) # flatten all dimensions except batch
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    #x = self.fc4(x)
    return x
</code></pre>
<p>net = Net()</p>
<p>Then I load the model and optimizer and also add lines<br>
net.fc4 = nn.Linear(10, 10)<br>
optimizer.add_param_group(<br>
{‘params’: [net.fc4.weight]})</p>
<p>Then I train the this model a few steps and save the state of model and optimizer.<br>
After that I want to continue the model training.<br>
So I am loading the new model with fc4 linear layer already in the model. The model loaded succesfully.<br>
But the optimizer throws an error</p>
<p>[ValueError: loaded state dict has a different number of parameter groups]</p>

MinqiuSun: <p>Have you solved this problem yet? I also met a similar problem <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"></p>