Insert New Layer in the Middle of a Pre-trained Model

wangchust: <p>Help!!! Does anyone knows how to insert a new layer in the middel of a pre-trained model? e.g insert a new conv in the middle of Resnet’s bottelneck.</p>

Kaixhin: <p>There’s no easy way to insert a new layer in the middle of an existing model as far as I’m aware. A definite solution is to build the structure that you want in a new class and then copy the corresponding weights over from the pretrained model.</p>

wangchust: <p>Figured!!!<br>
After loading model，we can directly specify model.conv_x = nn.Sequential([new_layer, model.conv_x]), by this way, we can still use thepretrained model.conv_x</p>

shirishr: <p><a class="mention" href="/u/wangchust">@wangchust</a><br>
Can you help a newbie like me. I import pretrained resnest34 as:</p>
<p>resnet = models.resnet34(pretrained=True)</p>
<p>Now I want to insert a conv2d 1x1 kernel layer before the fc to increase channel size from 512 to 6000 and then add a fc 6000 x 6000</p>
<p>I am so new to pytorch that I need some hand-holding. Can you write the lines of code needed?  I am still at monkey-see monkey-learn stage. Thanks in anticipation</p>

ptrblck: <p>I think the easiest approach would be to derive from <code>ResNet</code> and add your layers.<br>
This should do what you need:</p>
<pre><code class="lang-auto">
class MyResnet2(models.ResNet):
    def __init__(self, block, layers, num_classes=1000):
        super(MyResnet2, self).__init__(block, layers, num_classes)
        self.conv_feat = nn.Conv2d(in_channels=512, 
                                   out_channels=6000, 
                                   kernel_size=1)
        self.fc = nn.Linear(in_features=6000,
                            out_features=6000)
        
    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = self.conv_feat(x)
        x = x.view(x.size(0), -1)
        x = self.fc(x)

        return x

from torchvision.models.resnet import BasicBlock
model = MyResnet2(BasicBlock, [3, 4, 6, 3], 1000)
x = Variable(torch.randn(1, 3, 224, 224))
output = model(x)
</code></pre>
<p>Note that the shape of <code>x</code> in <code>x= self.avgpool(x)</code> is already <code>[batch, 512, 1, 1]</code> for <code>x.shape = [batch, 3 ,224, 224]</code>.<br>
You could therefore flatten <code>x</code> and just use two <code>Linear</code> layers, since it would be the same as a <code>Conv2d</code> with <code>kernel_size=1</code>.</p>

shirishr: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you sir!,<br>
Now if I re-read some of the tutorials, they will register in my head.</p>

wangchust: <p>Sorry for being so late  to reply. My way should be replace ‘Resnet.fc = nn.Linear(512, num_classes)’ with ‘Resnet.fc = nn.Sequential(nn.Conv2d(512, 6000), nn.Linear(6000,6000))’</p>

ptrblck: <p>Using this approach you would have to define a <code>Flatten</code> layer, since it will crash in the <code>Sequential</code> model.</p>
<pre><code class="lang-auto">class Flatten(nn.Module):
    def __init__(self):
        super(Flatten, self).__init__()
        
    def forward(self, x):
        x = x.view(x.size(0), -1)
        return x
        

seq = nn.Sequential(nn.Conv2d(512, 6000, 1),
                    Flatten(),
                    nn.Linear(6000,6000))

x = Variable(torch.randn(1, 512, 1, 1))
seq(x)
</code></pre>

shirishr: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/wangchust">@wangchust</a>,</p>
<p>Thank you both.</p>

Ritvik_Rawat: <p>I find this very useful to create a custom model which inserts a new layer in middle of ResNet, but how do we get the pretrained weights? This is giving me a new neural net which doesn’t have the pre trained weights.<br>
Thank you.</p>

ANKUR_GUPTA2: <p>I want to add attention layers in the pretrained resnet model how can I do so after every resnet block in the model.</p>

kendreaditya: <p>This is what I did, I think it works.</p>
<pre><code class="lang-auto">vgg11.features[0] = nn.Sequential(inserted_layer, vgg11.features[0])
</code></pre>
<p>I checked both parameters, and I think the pretrained parameters are preserved.</p>

Flock1: <p>I first divided the model into two parts; 1st half are the layers before the layer you want to add and the 2nd half contains the layers after your layer. So something like this:</p>
<pre><code class="lang-auto">encoder = nn.Sequential(*list(m.children())[:8])
decoder  = nn.Sequential(*list(m.children())[8:])
</code></pre>
<p>Then I added the layers I need, get a list of the layers and <code>append</code> each layer in the 2nd layer into the 1st one. Like this:</p>
<pre><code class="lang-auto">tmp_1 = list(encoder.children())
tmp_2 = list(decoder.children())
for i in tmp_2:
    tmp_1.append(i)
</code></pre>
<p>And then <code>model = nn.Sequential(*list(tmp_1))</code></p>

suresh_thommandru: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="12414">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">class MyResnet2(models.ResNet):
    def __init__(self, block, layers, num_classes=1000):
        super(MyResnet2, self).__init__(block, layers, num_classes)
        self.conv_feat = nn.Conv2d(in_channels=512, 
                                   out_channels=6000, 
                                   kernel_size=1)
        self.fc = nn.Linear(in_features=6000,
                            out_features=6000)
        
    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = self.conv_feat(x)
        x = x.view(x.size(0), -1)
        x = self.fc(x)

        return x

from torchvision.models.resnet import BasicBlock
model = MyResnet2(BasicBlock, [3, 4, 6, 3], 1000)
x = Variable(torch.randn(1, 3, 224, 224))
output = model(x)
</code></pre>
</blockquote>
</aside>
<p>Sir, In this, I would like to use the pre-trained weights up to layer4. And I have to do transfer learning for the remaining layers.</p>
<p>My requirement is<br>
// considering pretrained model<br>
from torchvision import models<br>
resnet50 = models.resnet50(pretrained = True)</p>
<p>//pruning the layers<br>
prune.l1_unstructured(resnet50.conv1, name=‘weight’, amount=0.4)</p>
<p>// adding layer in the middle (after layer 4)<br>
class MyResNet50(models.ResNet):</p>
<pre><code>def __init__(self, my_pretrained_model):

    super(MyResNet50, self).__init__(my_pretrained_model)
    self.pretrained_model = my_pretrained_model
    self.conv_feat = nn.Conv2d(in_channels=512, 
                               out_channels=512, 
                               kernel_size=3, padding=1)
    self.my_new_layers = nn.Sequential(nn.Linear(1000, 512),nn.Linear(512,1))
    
def forward(self, x):

    x = self.pretrained_model(x)
    x = self.conv_feat(x)
    x = self.avgpool(x)
    x = self.my_new_layers(x)

    return x
</code></pre>
<p>//creating object for modified resnet50<br>
myextendedmodel = MyResNet50(resnet50)</p>
<p>sir, is it possible to get modified resnet50?</p>

ptrblck: <aside class="quote no-group" data-username="suresh_thommandru" data-post="17" data-topic="12414">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/suresh_thommandru/48/45908_2.png" class="avatar"> suresh_thommandru:</div>
<blockquote>
<p>Sir, In this, I would like to use the pre-trained weights up to layer4. And I have to do transfer learning for the remaining layers.</p>
</blockquote>
</aside>
<p>Would this mean the overall model architecture stays the same as in the original ResNet while you are only training some layers (starting from layer5)? If so, you wouldn’t have to add new layers but would need to freeze the initial 4 layers by setting the <code>.requires_grad</code> attribute of the corresponding parameters to <code>False</code>.</p>

suresh_thommandru: <p>Thank you, sir for the solution.</p>

Hamze_Asadi: <p>Hi ptrblck</p>
<p>I have a question regarding loading a pretrained model,<br>
I have a pretrained model with some convolution layers (2d) that bias is disabled (bias flag in nn.Conv2d() is set to False), I was wondering if there is way to set the bias for this conv layer to true and initialize it the way we want?<br>
Thanks in advance</p>

ptrblck: <p>You could use the <code>strict=False</code> argument in <code>load_state_dict</code> as seen here:</p>
<pre><code class="lang-python">ref = nn.Conv2d(3, 3, 3, bias=False)
conv = nn.Conv2d(3, 3, 3, bias=True)

conv.load_state_dict(ref.state_dict())
# RuntimeError: Error(s) in loading state_dict for Conv2d:
# 	Missing key(s) in state_dict: "bias". 

conv.load_state_dict(ref.state_dict(), strict=False)
# _IncompatibleKeys(missing_keys=['bias'], unexpected_keys=[])
</code></pre>
<p>but make sure only the expected keys are missing by checking the returned object.</p>

Hamze_Asadi: <p>Thank you so much<br>
I use the following procedure and it seems is working:</p>
<pre><code class="lang-auto"> for name, layer in model.named_modules():
        if isinstance(layer, nn.Conv2d):
            if layer.bias is None:
                layer.bias = nn.Parameter(data=torch.Tensor(torch.randn(size=(layer.weight.shape[0],))/1e+10))
</code></pre>
<p>but your way is more clean.</p>