How to perform finetuning in Pytorch?

avijit_dasgupta: <p>Can anyone tell me how to do finetuning in pytorch? Suppose, I have loaded the Resnet 18 pretrained model. Now I want to finetune it on my own dataset which contain say 10 classes. How to remove the last output layer and change to as per my requirement?</p>

apaszke: <p>You can find an example at the bottom of <a href="http://pytorch.org/docs/notes/autograd.html#excluding-subgraphs-from-backward">this section</a> of autograd mechanics notes.</p>

avijit_dasgupta: <p>Thanks for your reply.</p>
<p>I have another doubt here. While fine-tuning, should we not use a smaller learning rate for the whole network (<a href="http://cs231n.github.io/transfer-learning/" rel="nofollow noopener">http://cs231n.github.io/transfer-learning/</a>)? Here the example says us to fix the network and only learn the weights in the newly added layers? Which one in recommended?</p>

apaszke: <p>In the snippet I’ve sent you only the last layer is optimized. The rest will be frozen (base parameters will receive no gradients - they have <code>requires_grad=False</code>). This is as if you used learning rate of 0 for that part. If you want to use a lower lr for the base see <a href="http://pytorch.org/docs/optim.html#per-parameter-options">this section</a> of the optim docs.</p>

avijit_dasgupta: <p>Thanks <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> It helped.</p>

avijit_dasgupta: <pre><code>optimizer = torch.optim.SGD([
            {'params': model.conv1.parameters()},
            {'params': model.bn1.parameters()},
            {'params': model.relu.parameters()},
            {'params': model.maxpool.parameters()},
            {'params': model.layer1.parameters()},
            {'params': model.layer2.parameters()},
            {'params': model.layer3.parameters()},
            {'params': model.layer4.parameters()},
            {'params': model.avgpool.parameters()},
            {'params': model.fc.parameters(), 'lr': opt.lr}
        ], lr=opt.lr*0.1, momentum=0.9)
</code></pre>
<p>Is this the correct way of defining  different learning rate to different layers of ResNet18? Or is there any other optimized way to do that?</p>

apaszke: <p>This should do it:</p>
<pre><code class="lang-python">ignored_params = list(map(id, model.fc.parameters()))
base_params = filter(lambda p: id(p) not in ignored_params,
                     model.parameters())

optimizer = torch.optim.SGD([
            {'params': base_params},
            {'params': model.fc.parameters(), 'lr': opt.lr}
        ], lr=opt.lr*0.1, momentum=0.9)

</code></pre>

avijit_dasgupta: <p>Great! Thanks <img src="//discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"></p>

zhoubinxyz: <p>This may also help to learn how to modify layers without changing other layers’ parameters and construct a new model.</p>
<pre><code class="lang-python">    model = models.vgg16(pretrained=True)
    print list(list(model.classifier.children())[1].parameters())
    mod = list(model.classifier.children())
    mod.pop()
    mod.append(torch.nn.Linear(4096, 2))
    new_classifier = torch.nn.Sequential(*mod)
    print list(list(new_classifier.children())[1].parameters())
    model.classifier = new_classifier
</code></pre>
<p>Also, you may consider to add pretrained models for VGG, which you may found here:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/vision/issues/28" target="_blank" rel="nofollow noopener">github.com/pytorch/vision</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/vision/issues/28" target="_blank" rel="nofollow noopener">Pretrained VGG models</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-01-18" data-time="22:26:53" data-timezone="UTC">10:26PM - 18 Jan 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2017-02-21" data-time="23:01:55" data-timezone="UTC">11:01PM - 21 Feb 17 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/jcjohnson" target="_blank" rel="nofollow noopener">
          <img alt="jcjohnson" src="https://avatars2.githubusercontent.com/u/2718714?v=4" class="onebox-avatar-inline" width="20" height="20">
          jcjohnson
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Is there any plan for pretrained VGG models, especially VGG16 or VGG19? These are needed for style transfer; AlexNet and ResNets...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>As for finetuning resnet, it is more easy:</p>
<pre><code class="lang-python">model = models.resnet18(pretrained=True)
model.fc = torch.nn.Linear(2048, 2)
</code></pre>

srv902: <p>How do I add new layers to existing pretrained models? Here, the last layer by <strong>name</strong> is replaced with a <strong>Linear</strong> layer. How to add another layer after that?</p>

apaszke: <pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, pretrained_model):
        self.pretrained_model = pretrained_model
        self.last_layer = ... # create layer

    def forward(self, x):
        return self.last_layer(self.pretrained_model(x))

pretrained_model = torchvision.models.resnet18(pretrained=True)
model = MyModel(pretrained_model)
</code></pre>

srv902: <p>Thank you for the help.</p>
<p>Another question, I am not able to see <strong>classifier</strong> attribute when loading a pretrained ResNet-18. However, I can directly access the <strong>children</strong> attribute without needing <strong>classifier</strong>. Is the <strong>classifier</strong> attribute part of old version of PyTorch or something I am missing?</p>

apaszke: <p>ResNets don’t have a <code>classifier</code> attribute. They use only a single fully-connected layer and it’s available as their <code>fc</code> attribute.</p>

panovr: <p><a class="mention" href="/u/zhoubinxyz">@zhoubinxyz</a> may you add a more complete example to illustrate how to fine-tuning VGG model on a custom dataset using PyTorch?</p>

apaszke: <p>You can get the basic idea form <a href="https://github.com/pytorch/examples/pull/58">this PR</a>.</p>

panovr: <p>There is an option named <code>--pretrained</code> in the imagenet <code>main.py</code> file. May I ask that if I use this option with a custom dataset like these:<br>
<code>python main.py --arch=alexnet --pretrained my_custom_dataset</code><br>
What will happen with this command? It seems that like a fine-tuning.</p>

apaszke: <p>I think <code>--pretrained</code> is meant for evaluation mode. The script doesn’t support finetuning at the moment.</p>

panovr: <aside class="quote no-group" data-username="apaszke" data-post="12" data-topic="419">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar"> apaszke:</div>
<blockquote>
<p>ast_laye</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/apaszke">@apaszke</a> I reference this <a href="https://github.com/pytorch/examples/pull/58" rel="noopener nofollow ugc">PR</a> for fine-tuning. For alexnet and vggnet, the original code replay all the fully-connected layers. May I ask:</p>
<ul>
<li>how can I only replace the last fully-connected layer for fine-tuning and freeze other fully-connected layers?</li>
<li>Is the <code>forward</code> the right way to code? Because you give some reference code above:</li>
</ul>
<blockquote>
<p>def forward(self, x):<br>
return self.last_layer(self.pretrained_model(x))</p>
</blockquote>
<p>Original fine-tuing code:</p>
<pre><code>class FineTuneModel(nn.Module):
    def __init__(self, original_model, arch, num_classes):
        super(FineTuneModel, self).__init__()

        if arch.startswith('alexnet') :
            self.features = original_model.features
            self.classifier = nn.Sequential(
                nn.Dropout(),
                nn.Linear(256 * 6 * 6, 4096),
                nn.ReLU(inplace=True),
                nn.Dropout(),
                nn.Linear(4096, 4096),
                nn.ReLU(inplace=True),
                nn.Linear(4096, num_classes),
            )
            self.modelName = 'alexnet'
        elif arch.startswith('resnet') :
            # Everything except the last linear layer
            self.features = nn.Sequential(*list(original_model.children())[:-1])
            self.classifier = nn.Sequential(
                nn.Linear(512, num_classes)
            )
            self.modelName = 'resnet'
        elif arch.startswith('vgg16'):
            self.features = original_model.features
            self.classifier = nn.Sequential(
                nn.Dropout(),
                nn.Linear(25088, 4096),
                nn.ReLU(inplace=True),
                nn.Dropout(),
                nn.Linear(4096, 4096),
                nn.ReLU(inplace=True),
                nn.Linear(4096, num_classes),
            )
            self.modelName = 'vgg16'
        else :
            raise("Finetuning not supported on this architecture yet")

        # Freeze those weights
        for p in self.features.parameters():
            p.requires_grad = False


    def forward(self, x):
        f = self.features(x)
        if self.modelName == 'alexnet' :
            f = f.view(f.size(0), 256 * 6 * 6)
        elif self.modelName == 'vgg16':
            f = f.view(f.size(0), -1)
        elif self.modelName == 'resnet' :
            f = f.view(f.size(0), -1)
        y = self.classifier(f)
        return y
</code></pre>

apaszke: <p><a href="https://discuss.pytorch.org/t/how-to-extract-features-of-an-image-from-a-trained-model/119/3?u=apaszke">This post</a> should help you.</p>

melgor: <p>I added following lines to imagenet example, using pretrained model of resnet18.</p>
<pre><code> for param in model.parameters():
      param.requires_grad = False

 # Replace the last fully-connected layer
 # Parameters of newly constructed modules have requires_grad=True by default
 model.fc = torch.nn.Linear(512, 3)

 optimizer = torch.optim.SGD(model.fc.parameters(), args.lr,
                            momentum=args.momentum,
                            weight_decay=args.weight_decay)
</code></pre>
<p>But then I have following error:</p>
<pre><code>File "main.py", line 234, in train
    loss.backward()
File "/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
RuntimeError: there are no graph nodes that require computing gradients
</code></pre>
<p>I would like to freeze all parameters of original ResNet18 and just learn the last layer with 3 classes. How I should do this correctly? Based on information from the forum, this should we the working version.</p>