How to delete layer in pretrained model?

EonianSky: <p>I am using the pre-trained model of vgg16 through torchvision. Now I don’t need the last layer (FC) in the network. How should I remove it?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cabb0bf7b1ca07d334c019b7ebcedb788abff80b.png" data-download-href="https://discuss.pytorch.org/uploads/default/cabb0bf7b1ca07d334c019b7ebcedb788abff80b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cabb0bf7b1ca07d334c019b7ebcedb788abff80b.png" alt="image" data-base62-sha1="sVrc4XYXal0AIno1bvEVdYIGQY3" width="540" height="500" data-dominant-color="F4F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">613×567 42.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Don’t you need this layer at all, i.e. do you want to get the <code>avgpool</code> back from your model?<br>
If so, you could createn a module returning its input:</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()
        
    def forward(self, x):
        return x


model = models.resnet18(pretrained=False)
model.fc = Identity()
x = torch.randn(1, 3, 224, 224)
output = model(x)
print(output.shape)
</code></pre>
<p>From the code it looks like you are using a resnet, so I used it in my examples.</p>
<p>However, usually you would like to use another linear layer with your number of classes as its output.</p>
<pre><code class="lang-auto">model.fc = nn.Linear(512, num_classes)
</code></pre>

EonianSky: <p>Thank you very much. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
PS:In my example, what is needed is the output of avgpool, no need for FC at all.</p>

EonianSky: <p>I found I put a wrong figure… If I use vgg16 and want to delete some layers,what should I do?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a9b4dd060ae58910d6c3ba26b3020542db401c03.png" alt="image" data-base62-sha1="odibOgCF2NVdgkOMvZ300saYAKf" width="406" height="195"></p>

ptrblck: <p>In this case you could use the following code:</p>
<pre><code class="lang-auto">model.classifier = nn.Sequential(*[model.classifier[i] for i in range(4)])
print(model.classifier)
</code></pre>
<p>EDIT:<br>
Alternatively, you can also call <code>.children</code>, since the range indexing might be cumbersome for large sequential models.</p>
<pre><code class="lang-auto">model.classifier = nn.Sequential(*list(model.classifier.children())[:-3])
</code></pre>

cad8bd801dfbc87c5d0f: <aside class="quote no-group quote-modified" data-username="EonianSky" data-post="1" data-topic="17648">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eoniansky/48/9538_2.png" class="avatar"> EonianSky:</div>
<blockquote>
<p>model.classifier = nn.Sequential(*list(model.classifier.children())[:-3])</p>
</blockquote>
</aside>
<p>Sorry but why did my model say ‘ResNet’ object has no attribute ‘classifier’</p>

lxtGH: <p>Try to use .fc attribute  Because Resnet doesn’t have classifier (with dropout in it), the above is vgg net <a class="mention" href="/u/cad8bd801dfbc87c5d0f">@cad8bd801dfbc87c5d0f</a></p>

Shihab_Shahriar: <p>using this approach however, I found specifying different learning for different layers in optimizer quite hard, which is very common in transfer learning. Can you please show how to achieve that in simpler ways? Thanks</p>

ptrblck: <p>Could you post your model architecture and which learning rates you would use for which layers?</p>

Shihab_Shahriar: <p>In resnet18, I wanted to simply replace ‘fc’ final layer, something like:<br>
<code>model.fc = nn.Linear(512,10)</code></p>
<p>Now if I wanted to do:</p>
<pre><code class="lang-auto">param_groups = [
    {'params':model.fc.parameters(),'lr':.001},
    {'params':model.others.parameters(),'lr':.0001},
]
optimizer = Adam(param_groups)
</code></pre>
<p>Finding <code>model.others.parameters()</code> part seemed hard. I could probably replace it with <code>params</code> array below:</p>
<pre><code class="lang-auto">params = []
for child in list(model.children())[:-1]:
    params.extend(list(child.parameters()))
</code></pre>
<p>Or define a class overriding <code>nn.Module</code>, which I did in the end. I was just wondering if there’s any elegant one liner, nothing big deal Thanks.</p>

ptrblck: <p>Your current code looks good!<br>
If the “filtering” would be a bit complicated, you could use Python <code>filter</code> to remove the <code>model.fc</code> layer by name, but in your use case I’m not sure there is a faster or more elegant way of passing <code>others</code> to the optimizer.</p>

ki2rin: <p>‘classifier’ is a key of the layer.</p>
<blockquote>
<p>(classifier): Sequential(<br>
…<br>
)</p>
</blockquote>
<p>So, you can refer to a sequence of a specific layer by its key.<br>
Perhaps, ResNet has a different key name to VGG in its implementation.<br>
<a class="mention" href="/u/cad8bd801dfbc87c5d0f">@cad8bd801dfbc87c5d0f</a></p>

ghazal_sahebzamani: <p>Sorry if this question seems trivial; why can’t we simply use:<br>
<code>model.classifier=model.classifier[:-1]</code><br>
Why is the Sequential line necessary?</p>

ptrblck: <p>Your assignment might actually work and I’m not sure, if the <code>nn.Sequential</code> wrapper was necessary when I’ve written the code.<br>
I think I just wanted to make sure the modules don’t get messed up somehow. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"><br>
Anyway, I’ve tested your line of code using <code>vgg16</code> and it seems to work perfectly, so you can just stick to your approach.</p>

marwa: <p>I tried to delete only the avgpool of the resnet but i got always mismatch errors. To avoid this error I replaced<br>
resnet.avgpool = nn.AvgPool2d(kernel_size=7, stride=1, padding=0)<br>
by<br>
resnet.avgpool = nn.AvgPool2d(kernel_size=1, stride=1, padding=0)<br>
and then<br>
resnet.fc = nn.Linear(512x7x7,number_of_classes).<br>
It is not the optimized way to deal with the problem. Do you have a better solution please?</p>

ptrblck: <p><code>nn.AvgPool2d</code> with a kernel size of 1 acts like an identity layer, your model basically got rid of the pooling layer.<br>
Alternatively, you could just define the <code>resnet.avgpool</code> as a “real” custom <code>Identity</code> layer:</p>
<pre><code class="lang-python">class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()
        
    def forward(self, x):
        return x

model.avgpool = Identity()
model.fc = nn.Linear(512*7*7, nb_classes)
</code></pre>
<p>Both approaches will yield the same result.</p>

Ivamcoder: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="17" data-topic="17648">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>class Identity(nn.Module): def <strong>init</strong>(self): super(Identity, self).<strong>init</strong>() def forward(self, x): return x</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>If we want to delete some sequenced layers in pretrained model, How could we do? for example in <code>renet</code> assume that we just want first three layers with fixed weights and omit the rest, I should put <code>Identity</code> for all layers I do not want? not any other way?</p>

ptrblck: <p>Not necessarily.<br>
If you would like to keep the <code>forward</code> method without overriding it, replacing a few layers with <code>nn.Identity</code> layers might be the fastest approach.<br>
However, if you would like to just use a few specific layers, I would recommend to override the class and write your custom model or alternatively reuse these layers in your custom model by passing them to your model.</p>

pablo_sanchez: <p>Dear<br>
Thank you very much for your help with the layer removal information, this helped me to fine tune in squeezenet.<br>
Regards</p>

Saharkakavand: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
is there any way that I can remove specific filters in pretrained model on vgg16(the base of faster rcnn) ? I want to prune vgg16 filter wise, so I need to remove specific filters in a layer and as a consequence in the next layer too.<br>
thanks</p>