Using portion of pretrained model

Vatsal_Malaviya: <p>I am trying to use the first few layers of trained resnet18, can anyone please help me with this?<br>
I got an error while using .features, I assume it may have depreciated is there any other way to do so?</p>
<p>And also can anyone please help why the following error is showing up<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c691003f499bf675e3b671f04961822f272b4c0.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c691003f499bf675e3b671f04961822f272b4c0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c691003f499bf675e3b671f04961822f272b4c0.png" alt="image" data-base62-sha1="8CpK6SlgM87zZ8johPbx20Z8RP2" width="690" height="189" data-dominant-color="462D26"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">972×267 16.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import torch
from torchvision import transforms, models
from torchsummary import summary

print(summary(models.resnet18(pretrained=True),(3,224,224)))
</code></pre>

Nikronic: <p>Hi,</p>
<p>Deprecation is not an error, it is warning that says a function has been changed in newer builds that you need to change your code according to the new pattern.</p>
<p>This error says that input data is on CUDA but model is on CPU. If you want to get the model’s architecture, just use <code>print(model)</code>. Then change or extract the layer you want.<br>
The reason is that <code>.features</code> does not exist is that model does not have any layer called <code>features</code>.</p>
<p>It would be great to read the documentations, there is a lot of elaborate examples and explanation about almost everything. Here is the one for <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">transfer learning</a>.</p>
<p>Bests</p>

Vatsal_Malaviya: <p>Sorry for the confusion, but my main goal was to extract/crop the network to say block 3 [using the results of generic convs in my network], In <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">transfer learning tutorial</a> I couldn’t find a way to extract the specific layer, can you please just suggest me a way to get values crop to block 3, so that i can add my model using nn.Sequential() after the cropped version of pretrained model<br>
Appreciate your help</p>

Nikronic: <p>I think it can help:</p>
<pre><code class="lang-auto">class CustomResNet(nn.Module):
    def __init__(self, model):
        super(CustomResNet, self).__init__()
        self.model = model
        self.part = nn.Sequential(
            model.conv1,
            model.bn1,
            model.relu,
            model.maxpool,
            model.layer1,
            model.layer2,
            model.layer3
        )
    def forward(self, x):
        x = self.part(x)
        return x

model = torchvision.models.resnet18(pretrained=True)
x = torch.randn(1, 3, 224, 224)
my_model = CustomResNet(model=model)
my_model(x).shape
</code></pre>
<p>But if you see the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#finetuning-the-convnet" rel="noopener nofollow ugc">fine tuning section</a> of  transfer learning tutorial, you can see that <code>fc</code> is a specific layer in resnet18 which can be accessed simply by <code>model.fc</code>. You might ask what about other layers, in the first post I have mentioned that:</p>
<aside class="quote no-group" data-username="Nikronic" data-post="2" data-topic="84229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>If you want to get the model’s architecture, just use <code>print(model)</code></p>
</blockquote>
</aside>
<p>So, by printing this, you will find that resnet has been constructed using the layers I have used in the snippet in this post.</p>
<p>PyTorch is highly modular framework, that is why we need to read the docs thoroughly.</p>
<p>Bests</p>

Vatsal_Malaviya: <p>Thanks for your help</p>

yeatmeret: <p><strong>i am wondering is there something wrong in your code?</strong></p>
<p>let’s run the following code, which made few changes on your code:</p>
<p><strong>=======================================================</strong><br>
import torch<br>
import torchvision<br>
import torch.nn as nn</p>
<p>class CustomResNet(nn.Module):<br>
def <strong>init</strong>(self, model):<br>
super(CustomResNet, self).<strong>init</strong>()<br>
self.model = model<br>
self.part = nn.Sequential(<br>
model.layer1,<br>
model.layer2,<br>
model.layer3<br>
)<br>
def forward(self, x):<br>
x = self.part(x)<br>
return x</p>
<p>model = torchvision.models.resnet18(pretrained=True)<br>
x = torch.randn(1, 3, 224, 224)<br>
my_model = CustomResNet(model=model)<br>
<span class="hashtag">#my_model</span>(x).shape<br>
print(my_model)</p>
<p>=======================================================<br>
the result is:</p>
<p>CustomResNet(<br>
(model): ResNet(<br>
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)<br>
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)<br>
(layer1): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(layer2): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(downsample): Sequential(<br>
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)<br>
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(layer3): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(downsample): Sequential(<br>
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(layer4): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(downsample): Sequential(<br>
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)<br>
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))<br>
(fc): Linear(in_features=512, out_features=1000, bias=True)<br>
)<br>
(part): Sequential(<br>
(0): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(1): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(downsample): Sequential(<br>
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)<br>
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(2): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(downsample): Sequential(<br>
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
)<br>
)</p>
<p><strong>================================================================</strong><br>
<strong>the result shows us that the customresnet has the same architecture as the resnet18, the customresnet should have only 3 layers, not the other part of the resnet18 like conv1 or maxpool.</strong></p>

yeatmeret: <p>Oh, i guess i just made a mistake.</p>
<p>the code print(my_model)</p>
<p>just give me back what resnet18 define in the function <strong>str</strong>, because customresnet is derived from resnet18.</p>

Nikronic: <p>Hi,</p>
<p>Well, although the entire original model’s architecture is still there (<code>self.model</code>), but in fact, only <code>self.part</code> is being used when you make calls to <code>customresnet</code>, because only modules used in <code>forward</code> function will be used <em>for computation</em>.</p>
<aside class="quote no-group" data-username="Nikronic" data-post="4" data-topic="84229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<pre><code class="lang-auto">def forward(self, x):
    x = self.part(x)
    return x
</code></pre>
</blockquote>
</aside>
<p>If I define <code>forward</code> as below, then original resnet plus the new defined part <code>self.part</code> will be run (assuming no dimension/layer mismatch):</p>
<pre><code class="lang-auto">def forward(self, x):
    x = self.model(x) 
    x = self.part(x)
    return x
</code></pre>
<p>Of course you can get rid of old resnet model if you don’t need it which is actually better and cleaner to do by just removing <code>self.model</code>.</p>
<p>The reason is that a module (any class that subclasses <code>torch.nn.Module</code>) that is assigned as attribute i.e. <code>self.model = model</code> in our case, will be registered by PyTorch automatically as described per docs <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html" rel="noopener nofollow ugc">Module — PyTorch 1.11.0 documentation</a>. I.e.:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee4a51bda1cddd5aba79458e19363d4de935e227.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee4a51bda1cddd5aba79458e19363d4de935e227" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee4a51bda1cddd5aba79458e19363d4de935e227.png" alt="image" data-base62-sha1="y00RIbfXejcbIrETOYDPq2Jc4g7" width="640" height="499" data-dominant-color="212223"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">743×580 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Bests</p>

yeatmeret: <p>thanks a lot~<br>
that is very kind of you</p>