RuntimeError: Error(s) in loading state_dict for ResNet: 	Missing key(s) in state_dict: "ca.fc1.weight", "ca.fc2.weight", "sa.conv1.weight", "ca1.fc1.weight", "ca1.fc2.weight", "sa1.conv1.weight"

Yaqi_CAI: <p>I modified the resnet50 network by：<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca4236349ab4f1817d9540421a6fcf4c514338ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/ca4236349ab4f1817d9540421a6fcf4c514338ff" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca4236349ab4f1817d9540421a6fcf4c514338ff.png" alt="image" data-base62-sha1="sRgj2dNsRFV2MRkDWyQ7DswSRxt" width="690" height="300" data-dominant-color="F2EEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">857×373 5.91 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec82836087a5b978be5a89a52b422dd315f93850.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec82836087a5b978be5a89a52b422dd315f93850" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec82836087a5b978be5a89a52b422dd315f93850.png" alt="image" data-base62-sha1="xKgiZdQ6y1APPPyNbYOUYMWElvq" width="690" height="493" data-dominant-color="FAF7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">728×521 6.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
These are the two classes:</p>
<blockquote>
<p>class ChannelAttention(nn.Module):<br>
def <strong>init</strong>(self, in_planes, ratio=16):<br>
super(ChannelAttention, self).<strong>init</strong>()<br>
self.avg_pool = nn.AdaptiveAvgPool2d(1)<br>
self.max_pool = nn.AdaptiveMaxPool2d(1)</p>
<pre><code>    self.fc1   = nn.Conv2d(in_planes, in_planes // 16, 1, bias=False)
    self.relu1 = nn.ReLU()
    self.fc2   = nn.Conv2d(in_planes // 16, in_planes, 1, bias=False)

    self.sigmoid = nn.Sigmoid()

def forward(self, x):
    avg_out = self.fc2(self.relu1(self.fc1(self.avg_pool(x))))
    max_out = self.fc2(self.relu1(self.fc1(self.max_pool(x))))
    out = avg_out + max_out
    return self.sigmoid(out)
</code></pre>
<p>class SpatialAttention(nn.Module):<br>
def <strong>init</strong>(self, kernel_size=7):<br>
super(SpatialAttention, self).<strong>init</strong>()</p>
<pre><code>    assert kernel_size in (3, 7), 'kernel size must be 3 or 7'
    padding = 3 if kernel_size == 7 else 1

    self.conv1 = nn.Conv2d(2, 1, kernel_size, padding=padding, bias=False)
    self.sigmoid = nn.Sigmoid()

def forward(self, x):
    avg_out = torch.mean(x, dim=1, keepdim=True)
    max_out, _ = torch.max(x, dim=1, keepdim=True)
    x = torch.cat([avg_out, max_out], dim=1)
    x = self.conv1(x)
    return self.sigmoid(x)
</code></pre>
</blockquote>
<p>But when I run: <strong>model = models.resnet50(pretrained=True)</strong>, it came out the problem, I have tried several solutions but still do not know how to fix it?</p>
<pre><code class="lang-auto">&gt; RuntimeError                              Traceback (most recent call last)
&gt; &lt;ipython-input-32-1d0a3ace5f7c&gt; in &lt;module&gt;
&gt; ----&gt; 1 model = models.resnet50(pretrained=True)
&gt;       2 
&gt;       3 
&gt;       4 #freeze layers
&gt;       5 # for param in model.parameters():
&gt; 
&gt; D:\pythonana\envs\pytorch\lib\site-packages\torchvision\models\resnet.py in resnet50(pretrained, progress, **kwargs)
&gt;     319         pretrained (bool): If True, returns a model pre-trained on ImageNet
&gt;     320         progress (bool): If True, displays a progress bar of the download to stderr
&gt; --&gt; 321     """
&gt;     322     return _resnet('resnet50', Bottleneck, [3, 4, 6, 3], pretrained, progress,
&gt;     323                    **kwargs)
&gt; 
&gt; D:\pythonana\envs\pytorch\lib\site-packages\torchvision\models\resnet.py in _resnet(arch, block, layers, pretrained, progress, **kwargs)
&gt;     282         state_dict = load_state_dict_from_url(model_urls[arch],
&gt;     283                                               progress=progress)
&gt; --&gt; 284         #model.load_state_dict(torch.load(PATH),strict=False)
&gt;     285 
&gt;     286         model.load_state_dict(state_dict)
&gt; 
&gt; D:\pythonana\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in load_state_dict(self, state_dict, strict)
&gt;     828         if len(error_msgs) &gt; 0:
&gt;     829             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
&gt; --&gt; 830                                self.__class__.__name__, "\n\t".join(error_msgs)))
&gt;     831         return _IncompatibleKeys(missing_keys, unexpected_keys)
&gt;     832 
&gt; 
&gt; RuntimeError: Error(s) in loading state_dict for ResNet:
&gt; 	Missing key(s) in state_dict: "ca.fc1.weight", "ca.fc2.weight", "sa.conv1.weight", "ca1.fc1.weight", "ca1.fc2.weight", "sa1.conv1.weight".
</code></pre>

yash1994: <p>Hi,</p>
<p>You’ve modified ResNet50 and you are trying to load weights from pre-trained vanilla ResNet50 which is causing the error because the pre-trained network doesn’t have weights for your modified layers <code>ChannelAttention</code> and <code>SpatialAttention</code>. I guess you have made modifications to <code>torchvision</code> source code. Don’t make any changes to the source code and implement a custom network and then add weights to your custom model by iterating <code>state_dict</code> of pretrained ResNet50 from torchvision.</p>

Yaqi_CAI: <aside class="quote no-group" data-username="yash1994" data-post="2" data-topic="74756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yash1994/48/21611_2.png" class="avatar"> yash1994:</div>
<blockquote>
<p>Don’t make any changes to the source code and implement a custom network and then add weights to your custom model by iterating <code>state_dict</code> of pretrained ResNet50 from torchvision.</p>
</blockquote>
</aside>
<p>But I have to add the attention mechanism codes to chage the network and get the weights, so what can i do to solve it?</p>

yash1994: <p>Check the implementation of ResNet50 from torchvision and implement custom network something like:</p>
<pre><code class="lang-python">class ModifiedResNet50(nn.Module):
    def __init__(self,):
         pass
    def forward(input):
         pass
</code></pre>
<p>Then load weights from pretrained ResNet50 like given below:</p>
<pre><code class="lang-python">vanilla_resnet = models.resnet50(pretrained=True)
modified_resnet = ModifiedResNet50()

for k,v in modified_resnet.state_dict():
    if k in vanilla_resnet.state_dict():
        modified_resnet.state_dict[k] = vanilla_resnet.state_dict[k]
</code></pre>

Yaqi_CAI: <p>Well, I add the class ModifiedResNet50 in the resnet.py</p>
<blockquote>
<p>class ModifiedResNet50(nn.Module):<br>
def <strong>init</strong>(self, block, layers, num_classes=1000, zero_init_residual=False,<br>
groups=1, width_per_group=64, replace_stride_with_dilation=None,<br>
norm_layer=None):<br>
super(ResNet, self).<strong>init</strong>()<br>
if norm_layer is None:<br>
norm_layer = nn.BatchNorm2d<br>
self._norm_layer = norm_layer</p>
<pre><code>    self.inplanes = 64
    self.dilation = 1
    if replace_stride_with_dilation is None:
        # each element in the tuple indicates if we should replace
        # the 2x2 stride with a dilated convolution instead
        replace_stride_with_dilation = [False, False, False]
    if len(replace_stride_with_dilation) != 3:
        raise ValueError("replace_stride_with_dilation should be None "
                         "or a 3-element tuple, got {}".format(replace_stride_with_dilation))
    self.groups = groups
    self.base_width = width_per_group
    self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3,
                           bias=False)
    self.bn1 = norm_layer(self.inplanes)
    self.relu = nn.ReLU(inplace=True)
  
  
     # 1st attention mechanism
    self.ca = ChannelAttention(self.inplanes)
    self.sa = SpatialAttention()
  
    self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
    self.layer1 = self._make_layer(block, 64, layers[0])
    self.layer2 = self._make_layer(block, 128, layers[1], stride=2,
                                   dilate=replace_stride_with_dilation[0])
    self.layer3 = self._make_layer(block, 256, layers[2], stride=2,
                                   dilate=replace_stride_with_dilation[1])
    self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
                                   dilate=replace_stride_with_dilation[2])
  							   
  # 2nd attention mechanism
    self.ca1 = ChannelAttention(self.inplanes)
    self.sa1 = SpatialAttention()							   
  							   
    self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
    self.fc = nn.Linear(512 * block.expansion, num_classes)

    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
        elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
            nn.init.constant_(m.weight, 1)
            nn.init.constant_(m.bias, 0)

    # Zero-initialize the last BN in each residual branch,
    # so that the residual branch starts with zeros, and each residual block behaves like an identity.
    # This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677
    if zero_init_residual:
        for m in self.modules():
            if isinstance(m, Bottleneck):
                nn.init.constant_(m.bn3.weight, 0)
            elif isinstance(m, BasicBlock):
                nn.init.constant_(m.bn2.weight, 0)

def _make_layer(self, block, planes, blocks, stride=1, dilate=False):
    norm_layer = self._norm_layer
    downsample = None
    previous_dilation = self.dilation
    if dilate:
        self.dilation *= stride
        stride = 1
    if stride != 1 or self.inplanes != planes * block.expansion:
        downsample = nn.Sequential(
            conv1x1(self.inplanes, planes * block.expansion, stride),
            norm_layer(planes * block.expansion),
        )

    layers = []
    layers.append(block(self.inplanes, planes, stride, downsample, self.groups,
                        self.base_width, previous_dilation, norm_layer))
    self.inplanes = planes * block.expansion
    for _ in range(1, blocks):
        layers.append(block(self.inplanes, planes, groups=self.groups,
                            base_width=self.base_width, dilation=self.dilation,
                            norm_layer=norm_layer))

    return nn.Sequential(*layers)

def _forward_impl(self, x):
    # See note [TorchScript super()]
    x = self.conv1(x)
    x = self.bn1(x)
    x = self.relu(x)
  
  #update
    x = self.ca(x) * x
    x = self.sa(x) * x
  
  
    x = self.maxpool(x)

    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)
  
  #update
    x = self.ca1(x) * x
    x = self.sa1(x) * x

    x = self.avgpool(x)
    x = torch.flatten(x, 1)
    x = self.fc(x)

    return x

def forward(self, x):
    return self._forward_impl(x)
</code></pre>
</blockquote>
<p>but why the problem occurs?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30e8272770178a2bd4a8b0b82884adb45bdb284c.png" data-download-href="https://discuss.pytorch.org/uploads/default/30e8272770178a2bd4a8b0b82884adb45bdb284c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30e8272770178a2bd4a8b0b82884adb45bdb284c.png" alt="image" data-base62-sha1="6YEiQtnqIKU0BOrD6PbnrLpJ3nC" width="621" height="500" data-dominant-color="F2F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">802×645 9.92 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

yash1994: <p>Hi,</p>
<p>it seems you are still editing source code, don’t add ModifiedResNet50 to resnet.py. You write/implement your custom class in your script/notebook cell. import bottleneck layers from torchvision if needed. Also change <code>super(ResNet, self).init()</code> to <code>super(ModifiedResNet50, self).init()</code>. I can’t write the whole code here, i can suggest a direction (so direct copy paste wont work here), rest you have to implement. Hope you understand.</p>

Yaqi_CAI: <p>Okay, thank you very much! I would try it.</p>

Yaqi_CAI: <p>Hello, I have modified the resnet_cbam successfully. And the loop should be:<br>
<strong>for k in modified_resnet.state_dict():</strong><br>
if k in vanilla_resnet.state_dict():<br>
modified_resnet.state_dict[k] = vanilla_resnet.state_dict[k]</p>

sgaur: <p>Can you please comment on this post.</p>
<aside class="quote quote-modified" data-post="1" data-topic="87227">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/3da27b/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/implementing-squeeze-and-excitation-block-with-pretrained-resnet/87227">Implementing Squeeze and Excitation block with pretrained resnet</a> <a class="badge-category__wrapper " href="/c/vision/kornia/19"><span data-category-id="19" data-parent-category-id="5" data-drop-close="true" class="badge-category --has-parent" title="A tag used for users of Kornia library"><span class="badge-category__name">kornia</span></span></a>
  </div>
  <blockquote>
    Hi 
I am trying to implement SE block with any pretrained resnet architecture. 
self.encoder = torchvision.models.resnet34(pretrained=pretrained)

self.conv1 = nn.Sequential(
    self.encoder.conv1,
    self.encoder.bn1,
    self.encoder.relu
    SeBlock(64,2),
    nn.MaxPool2d(2, 2) )

self.conv2 = nn.Sequential(
    self.encoder.layer1,
    SeBlock(64,2),
    )

self.conv3 = nn.Sequential(
    self.encoder.layer2,
    SeBlock(128,2),
    )

self.conv4 = nn.Sequential(
    self.encoder.layer3,
…
  </blockquote>
</aside>