ResNet, Bottleneck, Layers, groups, width_per_group

Oscar_Rangel: <p>I was on the web looking for how to implement ResNet in Pytorch, and found this code, but I have couple of days looking for information on the web to explain how it works and I have not found anything, that is why I hopw to find information here of the parameters what are they and how  they are used, or documentations.<br>
Thanks.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torchvision.models as models

class MyResNeXt(models.resnet.ResNet):
    def __init__(self, training=True):
        super(MyResNeXt, self).__init__(block=models.resnet.Bottleneck,
                                        layers=[3, 4, 6, 3], 
                                        groups=32, 
                                        width_per_group=4)
        self.fc = nn.Linear(2048, 1)
</code></pre>
<ul>
<li>why do we init the ResNet with the Bottleneck, any docs?<br>
2.- layers [3, 4, 6, 3] this depends on the type of resnet? 18, 50, 101? etc<br>
3.- groups?<br>
4.- width <em>per</em> group, can we use different sizes? 8, 16, 32, etc?<br>
5.- self.fc = nn.Linear(2048, 1) is is binary, but why not 2 ? in Keras you can use 2 as the final output</li>
</ul>

ptrblck: <ol>
<li>
<p>The sublocks of the resnet architecture can be defined as <code>BasicBlock</code> or <code>Bottleneck</code> based on the used resnet depth. E.g. <a href="https://github.com/pytorch/vision/blob/e2573a71c093bd5f8fe5a312cc642cd80a54f719/torchvision/models/resnet.py#L228-L249">resnet18 and resnet32</a> use <code>BasicBlock</code>, while <a href="https://github.com/pytorch/vision/blob/e2573a71c093bd5f8fe5a312cc642cd80a54f719/torchvision/models/resnet.py#L260">resnet&gt;=50</a> use <code>Bottleneck</code>.</p>
</li>
<li>
<p>Yes. Your mentioned configuration would fit <code>resnet34</code> and <code>resnet50</code> as seen <a href="https://github.com/pytorch/vision/blob/e2573a71c093bd5f8fe5a312cc642cd80a54f719/torchvision/models/resnet.py#L248-L260">here</a>.</p>
</li>
<li>
<p><code>Bottleneck</code> layers support the <code>groups</code> argument to create grouped convolutions. (<a href="https://github.com/pytorch/vision/blob/e2573a71c093bd5f8fe5a312cc642cd80a54f719/torchvision/models/resnet.py#L89">line of code</a>)</p>
</li>
<li>
<p>Again, a ResNeXt-specific setup for the <code>Bottleneck</code> layer. You could try different values, but would most likely have to look into the paper, how these values would interact with the channels etc.</p>
</li>
<li>
<p>You could treat your binary classification use case as a 2 class multi-class classification use case, if you set the number of output features to 2.</p>
</li>
</ol>

Oscar_Rangel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Wow!! this is great support, I am coming from the world of Keras and Tensorflow, as I believe that Pytorch is much better.</p>
<p>couple of  questions:<br>
1.- why allmost all the sample code I see in the web the do it like this:<br>
resnet = models.resnet50(pretrained=True) and not like this other way? is there any advantage?</p>
<p>1.-  how can you find tune this type of model<br>
2.- can you concatenate models as you do in keras ?</p>
<p>Thanks a lot!</p>

ptrblck: <ol>
<li>
<p>Creating the <code>resnet50</code> in a single line of code with pretrained weights is quite convenient instead of writing a custom class. If you don’t want to change e.g. the forward pass or any other modules, you could just stick to the <code>torchvision.models</code>.</p>
</li>
<li>
<p>Have a look at the <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">this</a> or <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html">this</a> tutorial for an introduction to finetuning the models.</p>
</li>
<li>
<p>You can create the computation graph dynamically in any form you wish.<br>
E.g. if you want to feed the output of one model to another one, you can just write:</p>
</li>
</ol>
<pre><code class="lang-python">output = model1(x)
output = model2(output)
loss = criterion(output, target)
loss.backward()
optimizer.step()
</code></pre>
<p>Autograd will make sure to create the gradients in both models as long as you haven’t detached a tensor from the computation graph (e.g. by using numpy methods or calling <code>tensor.detach()</code>).</p>

Oscar_Rangel: <p>Love you man!!! finally I am learning Pytorch!!!</p>

Oscar_Rangel: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,  I am looking at this link <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">here</a> point # 2 but there is something I am missing,</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
# Here the size of each output sample is set to 2.
# Alternatively, it can be generalized to nn.Linear(num_ftrs, len(class_names)).
model_ft.fc = nn.Linear(num_ftrs, 2)
</code></pre>
<p>Then you create another model…</p>
<pre><code class="lang-auto">model_conv = torchvision.models.resnet18(pretrained=True)
for param in model_conv.parameters():
    param.requires_grad = False

# Parameters of newly constructed modules have requires_grad=True by default
num_ftrs = model_conv.fc.in_features
model_conv.fc = nn.Linear(num_ftrs, 2)
</code></pre>
<p>they have different model names and in the first one, you use all the layers except the last linear one and set it to 2.</p>
<p>In the second piece of code, you freeze all layers and then the rest is basically the same.</p>
<p>BUT I don’t see any fine-tuning here… at least not from Keras way of doing it… where we use the same model and here are different models.</p>
<p>you mean we have to do this, or what is that I am missing</p>
<pre><code class="lang-auto">output = model_ft(x)
output = model2(model_conv)
loss = criterion(output, target)
</code></pre>
<p>Thanks a lot for your help!</p>

ptrblck: <p>Maybe I’m using the wrong terminology, but by fine tuning a model I mean to use a pretrained model, make some necessary changes for the new dataset (e.g. number of output units) and train this model using the new dataset.</p>
<p>The tutorial explains two different approaches, where the first one trains all parameters, while the latter one only trains the last output layer.</p>
<p>Passing the output of one model to another one is independent from your fine tuning use case, so you can ignore it for now.</p>

Oscar_Rangel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ahh I understand now, it is completely different int keras, a lot easier in Pytorch hehe.</p>
<p>so when this person in the internet is declaring this way the resnet, he is using the finetune technique, right ?</p>
<pre><code class="lang-auto">class MyResNeXt(models.resnet.ResNet):
    def __init__(self, training=True):
        super(MyResNeXt, self).__init__(block=models.resnet.Bottleneck,
                                        layers=[3, 4, 6, 3], 
                                        groups=32, 
                                        width_per_group=4)
        self.fc = nn.Linear(2048, 1)
</code></pre>
<p>I was looking into the second link and was wondering how you go about instantiating is this correct? or where to find information of how to instantiate the ResNet 50 this way?</p>
<pre><code class="lang-auto">model_ft = models.ResNet50(block=models.resnet.Bottleneck,
                                        layers=[3, 4, 6, 3], 
                                        groups=32, 
                                        width_per_group=4)
        self.fc = nn.Linear(2048, 1)
</code></pre>
<p>Thanks again for your valuable and fast help!</p>
<p>PS.<br>
can you point me where to learn how to concatenate models in Pytorch, or how you call this technique?</p>
<pre><code class="lang-auto">output = model1(x)
output = model2(output)
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Oscar_Rangel" data-post="8" data-topic="69088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oscar_rangel/48/20363_2.png" class="avatar"> Oscar_Rangel:</div>
<blockquote>
<p>so when this person in the internet is declaring this way the resnet, he is using the finetune technique, right ?</p>
</blockquote>
</aside>
<p>Not necessarily. I refer to “fine tuning” as using pretrained parameters and train the model on another dataset.<br>
The author of the mentioned code snippet creates a new model called <code>MyResNeXt</code> by deriving from <code>models.resnet.ResNet</code> as the base model. This allows him to use all parent modules and might implementing his model easier.<br>
The author doesn’t have to use the pretrained parameters of the base class, so it’s unrelated to a fine tuning task.</p>
<aside class="quote no-group" data-username="Oscar_Rangel" data-post="8" data-topic="69088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oscar_rangel/48/20363_2.png" class="avatar"> Oscar_Rangel:</div>
<blockquote>
<p>where to find information of how to instantiate the ResNet 50 this way?</p>
</blockquote>
</aside>
<p>I would recommend to use the <a href="https://pytorch.org/docs/stable/torchvision/models.html"><code>torchvision.models</code></a>, if they fit your use case. Initializing the resnet “manually” as shown in your code snippet is needed, if you want to manipulate the model in a non-trivial way (e.g. change whole blocks inside the model etc.).</p>
<aside class="quote no-group" data-username="Oscar_Rangel" data-post="8" data-topic="69088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oscar_rangel/48/20363_2.png" class="avatar"> Oscar_Rangel:</div>
<blockquote>
<p>can you point me where to learn how to concatenate models in Pytorch, or how you call this technique?</p>
</blockquote>
</aside>
<p>I might have used the wrong wording again, but there is not really anything special about this approach.<br>
You are writing models in the same way by just passing the output of one layer to the next one:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 3, 1, 1)
        self.conv2 = nn.Conv2d(3, 6, 3, 1, 1)
        self.act = nn.ReLU()
        
    def forward(self, x):
        x = self.act(self.conv1(x))
        x = self.conv2(x)
        return x
</code></pre>
<p>In this example you are passing the output of the first convolution to the next one.<br>
Since <code>MyModel</code> derives from <code>nn.Module</code> it can be treated as all other <code>nn.Modules</code> e.g. <code>nn.Conv2d</code>.</p>

Oscar_Rangel: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I really appreciate your help and time.</p>

Oscar_Rangel: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, sorry <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=9" title=":innocent:" class="emoji" alt=":innocent:"> need your help again, I am trying to learn how to implement custom class for resnet50 &gt; models.</p>
<p>I am trying this but is giving me an error.</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-31-44bf8238479a&gt; in &lt;module&gt;
     13     checkpoint = torch.load("../models/resnext50_32x4d-7cdf4587.pth")
     14 
---&gt; 15     model_ft = MyResNeXt().to(device) # models.resnet18(pretrained=True)
     16 
     17     del checkpoint
 RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>here is the code</p>
<pre><code class="lang-auto">import torchvision.models as models

class MyResNeXt(models.resnet.ResNet):
    def __init__(self, training=True):
        super(MyResNeXt, self).__init__(block=models.resnet.Bottleneck,
                                        layers=[3, 4, 6, 3], 
                                        groups=32, 
                                        width_per_group=4)

        self.load_state_dict(checkpoint)

        # Override the existing FC layer with a new one.
        self.fc = nn.Linear(2048, 1)

def freeze_until(net, param_name):
    found_name = False
    for name, params in net.named_parameters():
        if name == param_name:
            found_name = True
        params.requires_grad = found_name

######################################################################
# Finetuning the convnet
# ----------------------
#
# Load a pretrained model and reset final fully connected layer.
#
# loop over the number of models to train
for i in np.arange(0, 10):
    
    # initialize the optimizer and model
    print("[INFO] training model {}/{}".format(i + 1, 10))
    
    checkpoint = torch.load("../models/resnext50_32x4d-7cdf4587.pth")
    
    model_ft = MyResNeXt().to(device) # models.resnet18(pretrained=True)
    
    del checkpoint
    
    freeze_until(model_ft, "layer4.0.conv1.weight")

    criterion = nn.CrossEntropyLoss()

    # Observe that all parameters are being optimized
    optimizer_ft = torch.optim.Adam(model_ft.parameters(), lr=1e-5)

    # Decay LR by a factor of 0.1 every 7 epochs
    exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
    
    # Train and evaluate
    model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25)
    
    # save the model to disk
    p = ["models", "model_resnet18_adam{}.model".format(i)]
    checkpoint = { "optimizer": optimizer_ft.state_dict(),"model": model_ft.state_dict() }
    torch.save(checkpoint, p)
</code></pre>
<p>Thanks again for your help!!! I really have been tryingto find advance info in the web about pytorch, but nothing reall that goes beyond MNIST…</p>

Oscar_Rangel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> found the solution on the forum. you and the forum are of great help!</p>
<p>“Sir, I have got the error. It was because i had two classes and was using 1 at the output layer, which is used in other frameworks. But as I changed it to 2 my code is running. Similar thing happened in the age data where total number of classes were 104 but the actual age was from 1-116 and I was f…”</p>
<p>But I though that we need to use 1 for two class classification. and two was for more than 2.</p>
<p>Thanks again,</p>

ptrblck: <aside class="quote no-group" data-username="Oscar_Rangel" data-post="12" data-topic="69088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oscar_rangel/48/20363_2.png" class="avatar"> Oscar_Rangel:</div>
<blockquote>
<p>But I though that we need to use 1 for two class classification. and two was for more than 2.</p>
</blockquote>
</aside>
<p>No. You can use a single output with e.g. <code>nn.BCEWithLogitsLoss</code> for a binary classification.<br>
Alternatively you could also use two outputs with <code>nn.CrossEntropyLoss</code> for a “two class classification”, which would also classify each sample to one of two classes.</p>
<p>Note that the latter approach will double the output units and thus the last weight matrix.</p>
<p>Besides using another loss function, the target would also be different.<br>
<code>nn.BCEWithLogitsLoss</code> expects the target as a <code>FloatTensor</code> with values in the range <code>[0, 1]</code>, while <code>nn.CrossEntropyLoss</code> expects it to be a <code>LongTensor</code> with class indices in the range <code>[0, num_classes-1]</code>.</p>

Oscar_Rangel: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I see and understand now!!! THANKS!!!</p>

Lilun_Cheng: <p>Thanks for the explanation</p>