RuntimeError: Error(s) in loading state_dict for DataParallel: Missing key(s) in state_dict

Jane1: <p>I’m new to PyTorch, working on some image processing-related projects. I’m currently using ResNet101 and here I am facing this problem:</p>
<p>RuntimeError: Error(s) in loading state_dict for DataParallel:<br>
Missing key(s) in state_dict: “module.context_encoding.stages.0.2.bn.weight”, “module.context_encoding.stages.0.2.bn.bias”, “module.context_encoding.stages<br>
.0.2.bn.running_mean”, “module.context_encoding.stages.0.2.bn.running_var”, “module.context_encoding.stages.1.2.bn.weight”, “module.context_encoding.stages.1.2.bn.<br>
bias”, “module.context_encoding.stages.1.2.bn.running_mean”, “module.context_encoding.stages.1.2.bn.running_var”, “module.context_encoding.stages.2.2.bn.weight”, "<br>
module.context_encoding.stages.2.2.bn.bias", “module.context_encoding.stages.2.2.bn.running_mean”, “module.context_encoding.stages.2.2.bn.running_var”, “module.con<br>
text_encoding.stages.3.2.bn.weight”, “module.context_encoding.stages.3.2.bn.bias”, “module.context_encoding.stages.3.2.bn.running_mean”, “module.context_encoding.s<br>
tages.3.2.bn.running_var”, “module.context_encoding.bottleneck.1.bn.weight”, “module.context_encoding.bottleneck.1.bn.bias”, “module.context_encoding.bottleneck.1.<br>
bn.running_mean”, “module.context_encoding.bottleneck.1.bn.running_var”, “module.edge.conv1.1.bn.weight”, “module.edge.conv1.1.bn.bias”, “<a href="http://module.edge.conv1.1.bn.ru" rel="noopener nofollow ugc">module.edge.conv1.1.bn.ru</a><br>
nning_mean”, “module.edge.conv1.1.bn.running_var”, “module.edge.conv2.1.bn.weight”, “module.edge.conv2.1.bn.bias”, “module.edge.conv2.1.bn.running_mean”, “module.e<br>
dge.conv2.1.bn.running_var”, “module.edge.conv3.1.bn.weight”, “module.edge.conv3.1.bn.bias”, “module.edge.conv3.1.bn.running_mean”, “module.edge.conv3.1.bn.running<br>
_var”, “module.decoder.conv1.1.bn.weight”, “module.decoder.conv1.1.bn.bias”, “module.decoder.conv1.1.bn.running_mean”, “module.decoder.conv1.1.bn.running_var”, “mo<br>
dule.decoder.conv2.1.bn.weight”, “module.decoder.conv2.1.bn.bias”, “module.decoder.conv2.1.bn.running_mean”, “module.decoder.conv2.1.bn.running_var”, “module.decod<br>
er.conv3.1.bn.weight”, “module.decoder.conv3.1.bn.bias”, “module.decoder.conv3.1.bn.running_mean”, “module.decoder.conv3.1.bn.running_var”, “module.decoder.conv3.3<br>
.bn.weight”, “module.decoder.conv3.3.bn.bias”, “module.decoder.conv3.3.bn.running_mean”, “module.decoder.conv3.3.bn.running_var”, “module.fushion.1.bn.weight”, “module.fushion.1.bn.bias”, “module.fushion.1.bn.running_mean”, “module.fushion.1.bn.running_var”.<br>
Unexpected key(s) in state_dict: “module.context_encoding.stages.0.2.weight”, “module.context_encoding.stages.0.2.bias”, “module.context_encoding.stages.0.<br>
2.running_mean”, “module.context_encoding.stages.0.2.running_var”, “module.context_encoding.stages.1.2.weight”, “module.context_encoding.stages.1.2.bias”, “module.<br>
context_encoding.stages.1.2.running_mean”, “module.context_encoding.stages.1.2.running_var”, “module.context_encoding.stages.2.2.weight”, “module.context_encoding.<br>
stages.2.2.bias”, “module.context_encoding.stages.2.2.running_mean”, “module.context_encoding.stages.2.2.running_var”, “module.context_encoding.stages.3.2.weight”,<br>
“module.context_encoding.stages.3.2.bias”, “module.context_encoding.stages.3.2.running_mean”, “module.context_encoding.stages.3.2.running_var”, “module.context_en<br>
coding.bottleneck.1.weight”, “module.context_encoding.bottleneck.1.bias”, “module.context_encoding.bottleneck.1.running_mean”, “module.context_encoding.bottleneck.<br>
1.running_var”, “module.edge.conv1.1.weight”, “module.edge.conv1.1.bias”, “module.edge.conv1.1.running_mean”, “module.edge.conv1.1.running_var”, “module.edge.conv2<br>
.1.weight”, “module.edge.conv2.1.bias”, “module.edge.conv2.1.running_mean”, “module.edge.conv2.1.running_var”, “module.edge.conv3.1.weight”, “module.edge.conv3.1.b<br>
ias”, “module.edge.conv3.1.running_mean”, “module.edge.conv3.1.running_var”, “module.decoder.conv1.1.weight”, “module.decoder.conv1.1.bias”, “module.decoder.conv1.<br>
1.running_mean”, “module.decoder.conv1.1.running_var”, “module.decoder.conv2.1.weight”, “module.decoder.conv2.1.bias”, “module.decoder.conv2.1.running_mean”, “modu<br>
le.decoder.conv2.1.running_var”, “module.decoder.conv3.1.weight”, “module.decoder.conv3.1.bias”, “module.decoder.conv3.1.running_mean”, “module.decoder.conv3.1.run<br>
ning_var”, “module.decoder.conv3.3.weight”, “module.decoder.conv3.3.bias”, “module.decoder.conv3.3.running_mean”, “module.decoder.conv3.3.running_var”, “module.fushion.1.weight”, “module.fushion.1.bias”, “module.fushion.1.running_mean”, “module.fushion.1.running_var”.</p>
<p>I have only little knowledge on PyTorch. Based on what I understand from the error messages, it seems like the bn layer is not loaded. Can anyone help me with this? Thanks!</p>

Jane1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I’ve seen you helping out others with their problems which are similar as mine. Could you please help? Thank you very very very very very much <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I guess you are trying to load a <code>state_dict</code> stored by a plain model into an <code>nn.DataParallel</code> model, as the <code>load_state_dict</code> method is complaining about the missing <code>.module</code> attributes.<br>
If so, then try to load the <code>state_dict</code> into the model before wrapping it into <code>nn.DataParallel</code>:</p>
<pre><code class="lang-python">model = MyModel()
model.load_state_dict(torch.load(path))
model = nn.DataParallel(model)
</code></pre>

Jane1: <p>Thanks for replying!</p>
<p>I have changed my code from:<br>
model = network(num_classes=num_classes, pretrained=None)<br>
model = nn.DataParallel(model)<br>
state_dict = torch.load(args.restore_weight)[‘state_dict’]<br>
model.load_state_dict(state_dict)<br>
model.eval()</p>
<p>To:<br>
model = network(num_classes=num_classes, pretrained=None)<br>
model = nn.DataParallel(model)<br>
state_dict = torch.load(args.restore_weight)[‘state_dict’]<br>
model.load_state_dict(torch.load(‘./resnet101-imagenet.pth’))[‘state_dict’]<br>
model.load_state_dict(state_dict)<br>
model.eval()</p>
<p>Could you please guide me if I’m doing it correctly?<br>
Now the runtime error is:<br>
RuntimeError: Error(s) in loading state_dict for DataParallel: Missing key(s) in state_dict: “module.conv1.weight”, “module.bn1.weight”, “module.bn1.bias”, “module.bn1.running_mean”,…<br>
Unexpected key(s) in state_dict: “conv1.weight”, “bn1.running_mean”, “bn1.running_var”, “bn1.weight”, “bn1.bias”, “layer1.0.conv1.weight”, “layer1.0.bn1.running_mean”,…</p>
<p>Now it says the “module.” is missing…</p>

ptrblck: <p>You are still wrapping the <code>model</code> into <code>nn.DataParallel</code> before loading the <code>state_dict</code> and the error message is thus the same.<br>
Swap the order: i.e. load the <code>state_dict</code> into the <code>model</code> and wrap it into <code>nn.DataParallel</code> afterwards.</p>

Jane1: <p>Now my code is:<br>
model = network(num_classes=num_classes, pretrained=None)<br>
state_dict = model.load_state_dict(torch.load(‘./resnet101-imagenet.pth’))<br>
model.load_state_dict(state_dict)<br>
model = nn.DataParallel(model)<br>
model.eval()</p>
<p>RuntimeError: Error(s) in loading state_dict for ResNet: Missing key(s) in state_dict: “context_encoding.stages.0.1.weight”, “context_encoding.stages.0.2.bn.weight”, “context_encoding.stages.0.2.bn.bias”, “context_encoding.stages.0.2.bn.running_mean”, “context_encoding.stages.0.2.bn.running_var”, “context_encoding.stages.1.1.weight”, “context_encoding.stages.1.2.bn.weight”, “context_encoding.stages.1.2.bn.bias”, “context_encoding.stages.1.2.bn.running_mean”, “context_encoding.stages.1.2.bn.running_var”, “context_encoding.stages.2.1.weight”, “context_encoding.stages.2.2.bn.weight”, “context_encoding.stages.2.2.bn.bias”, “context_encoding.stages.2.2.bn.running_mean”, “context_encoding.stages.2.2.bn.running_var”, “context_encoding.stages.3.1.weight”, “context_encoding.stages.3.2.bn.weight”, “context_encoding.stages.3.2.bn.bias”, “context_encoding.stages.3.2.bn.running_mean”, “context_encoding.stages.3.2.bn.running_var”, “context_encoding.bottleneck.0.weight”, “context_encoding.bottleneck.1.bn.weight”, “context_encoding.bottleneck.1.bn.bias”, “context_encoding.bottleneck.1.bn.running_mean”, “context_encoding.bottleneck.1.bn.running_var”, “edge.conv1.0.weight”, “edge.conv1.1.bn.weight”, “edge.conv1.1.bn.bias”, “edge.conv1.1.bn.running_mean”, “edge.conv1.1.bn.running_var”, “edge.conv2.0.weight”, “edge.conv2.1.bn.weight”, “edge.conv2.1.bn.bias”, “edge.conv2.1.bn.running_mean”, “edge.conv2.1.bn.running_var”, “edge.conv3.0.weight”, “edge.conv3.1.bn.weight”, “edge.conv3.1.bn.bias”, “edge.conv3.1.bn.running_mean”, “edge.conv3.1.bn.running_var”, “edge.conv4.weight”, “edge.conv4.bias”, “edge.conv5.weight”, “edge.conv5.bias”, “decoder.conv1.0.weight”, “decoder.conv1.1.bn.weight”, “decoder.conv1.1.bn.bias”, “decoder.conv1.1.bn.running_mean”, “decoder.conv1.1.bn.running_var”, “decoder.conv2.0.weight”, “decoder.conv2.1.bn.weight”, “decoder.conv2.1.bn.bias”, “decoder.conv2.1.bn.running_mean”, “decoder.conv2.1.bn.running_var”, “decoder.conv3.0.weight”, “decoder.conv3.1.bn.weight”, “decoder.conv3.1.bn.bias”, “decoder.conv3.1.bn.running_mean”, “decoder.conv3.1.bn.running_var”, “decoder.conv3.2.weight”, “decoder.conv3.3.bn.weight”, “decoder.conv3.3.bn.bias”, “decoder.conv3.3.bn.running_mean”, “decoder.conv3.3.bn.running_var”, “decoder.conv4.weight”, “decoder.conv4.bias”, “fushion.0.weight”, “fushion.1.bn.weight”, “fushion.1.bn.bias”, “fushion.1.bn.running_mean”, “fushion.1.bn.running_var”, “fushion.3.weight”, “fushion.3.bias”. Unexpected key(s) in state_dict: “fc.weight”, “fc.bias”.</p>
<p>Did I get you correctly for first loading the state_dict into model then only wrap it?</p>

ptrblck: <p>Yes, this was my suggestion. The error message now changed but your model is also unable to map this <code>state_dict</code> to the modules.<br>
You would have to check how the internal modules in your <code>model</code> are defined and in particular check what the <code>state_dict</code> contains.<br>
In particular, your <code>model</code> seems to contain <code>model.context_encoding</code>, <code>model.edge</code>, etc. which are all missing in the <code>state_dict</code>.</p>

Jane1: <p>Alright, no problem. Thanks a lot for your help! I will continue debugging the code and see what is the issue. Sincere thanks! Have a nice day <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Sure, let me know if you get stuck.<br>
PS: I would start by checking the module names of the model and compare it against the keys in the <code>state_dict</code>. Something like this could give you more information what’s causing the mismatch:</p>
<pre><code class="lang-python">model = models.resnet18()
state_dict = models.vgg16().state_dict()

for name, _ in model.named_modules():
    print(name)
    
for key in state_dict:
    print(key)
</code></pre>
<p>This would of course also create a mismatch when you are trying to load the <code>VGG16</code> <code>state_dict</code> into a <code>ResNet</code> and would show that the actual module names are completely different.</p>

Jane1: <p>Hi. With the below code:<br>
model = network(num_classes=num_classes, pretrained=None)<br>
state_dict = torch.load(args.restore_weight)[‘state_dict’]<br>
model.load_state_dict(state_dict)<br>
model = nn.DataParallel(model)<br>
model.eval()<br>
I face the problem of having an extra ‘module.’ in the keys for state_dict, while the ‘bn.’ in the keys is still present. So, I have thought of removing ‘module.’ with the below code:<br>
model = network(num_classes=num_classes, pretrained=None)<br>
state_dict = torch.load(args.restore_weight)[‘state_dict’]<br>
from collections import OrderedDict<br>
new_state_dict = OrderedDict()<br>
for k, v in state_dict.items():<br>
name = k[7:]  # remove <code>module.</code><br>
new_state_dict[name] = v<br>
model.load_state_dict(new_state_dict)<br>
model = nn.DataParallel(model)<br>
model.eval()<br>
With this, now I finally have ‘module.’ removed. However, another problem I face which is the ‘bn.’ is removed as well. Do you have any idea on this? Thanks in advance.</p>

ptrblck: <p>Remove the <code>.module</code> string from the key conditionally if it’s found only by adding an if condition before calling: <code>name = k[7:]</code> so that other parameter keys won’t be changed.</p>