PyTorch Transfer Learning Methodology

Mona_Jalal: <p><strong>How is PyTorch doing the transfer learning? Does it just take the last layer of resnet and pass it to a new neural network classifier for my categories or does it retrain all the way back to resnet?</strong></p>
<p>Also, apart from what is in <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">this link</a>, are there other implementation of Transfer Learning in PyTorch?</p>

ptrblck: <p>It depends on you and your use case, which part of the model will be fine tuned.</p>
<p><a href="http://cs231n.github.io/transfer-learning/">Andrej Karpathy explained in Stanford’s CS231</a> you can use the size and similarity of your data as a baseline:</p>
<blockquote>
<ol>
<li><em>New dataset is small and similar to original dataset</em> . Since the data is small, it is not a good idea to fine-tune the ConvNet due to overfitting concerns. Since the data is similar to the original data, we expect higher-level features in the ConvNet to be relevant to this dataset as well. Hence, the best idea might be to train a linear classifier on the CNN codes.</li>
<li><em>New dataset is large and similar to the original dataset</em> . Since we have more data, we can have more confidence that we won’t overfit if we were to try to fine-tune through the full network.</li>
<li><em>New dataset is small but very different from the original dataset</em> . Since the data is small, it is likely best to only train a linear classifier. Since the dataset is very different, it might not be best to train the classifier form the top of the network, which contains more dataset-specific features. Instead, it might work better to train the SVM classifier from activations somewhere earlier in the network.</li>
<li><em>New dataset is large and very different from the original dataset</em> . Since the dataset is very large, we may expect that we can afford to train a ConvNet from scratch. However, in practice it is very often still beneficial to initialize with weights from a pretrained model. In this case, we would have enough data and confidence to fine-tune through the entire network.</li>
</ol>
</blockquote>
<p>It worked out well most of the time for me.</p>

Mona_Jalal: <p>And can you please specify which one <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a> is using? Thanks for the response. I assume the link I am sharing is using finetuning and is not backproping the result of new last layers classification through the entire pre-trained resnet. Also, from what 1) says, it seems the link is not doing a good job? because we have small dataset and we are fine-tuning a convnet? The PyTorch tutorial on transfer learning and 1) now seems contradictory to me. Can you please explain if I am wrong?</p>

ptrblck: <p>In the tutorial two approaches are shown.<br>
The first part shows how to fine tune the whole model, while the second one uses the pre-trained model as a fixed feature extractor and only trains the new classifier (linear layer).</p>
<p>Well, I wouldn’t see it as not doing a good job. The tutorial is explaining all necessary steps using PyTorch to fine tune your model. The data, accuracy, etc. are not the main focus in my opinion.<br>
Sure, using a small dataset and natural images, you would most likely want to just fine tune the classifier (case 1).</p>

tumble-weed: <p>the tutorial is backpropagating through the complete network. You can see it replace the last <code>model_ft.fc</code> layer but using all the <code>model_ft.parameters()</code> in  <code>optimizer_ft</code>.<br>
It’s a tutorial, so its purpose is more to give a sense of pytorch functionality rather than achieve the SOA on any benchmark.</p>

monster: <p>Hi,<br>
I have trained model A by using transfer learning (FasterRCNN -pretrained model), now I want to use model A as a pretrained model but getting issues.</p>
<p>step1=FasterRCNN to model A<br>
step2= model A to model B</p>
<p>First I am not sure that it is a good approach or not, I am just trying it.</p>
<p>code sample issue</p>
<pre><code class="lang-auto">self._model=torch.load("modelA.pth",map_location=torch.device('cpu'))
in_features = self._model.roi_heads.box_predictor.cls_score.in_features
# Replace the pre-trained head with a new one (note: +1 because of the __background__ class)

</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">in_features = self._model.roi_heads.box_predictor.cls_score.in_features
AttributeError: 'collections.OrderedDict' object has no attribute 'roi_heads'
</code></pre>
<p>Please let me know what I am doing wrong and how to do it.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

ptrblck: <p>Based on the error message it seems <code>torch.load('modelA.pth')</code> loads a <code>state_dict</code> (which is an <code>OrderedDict</code>).<br>
If that’s the case (which is also the recommended way of loading a model), you would have to create the model object first and then load the <code>state_dict</code> as seen here:</p>
<pre><code class="lang-python">model = FasterRCNN()
state_dict = torch.load('modelA.pth', map_location='cpu')
model.load_state_dict(state_dict)
in_features = model.roi_heads...
</code></pre>

beto16: <p>If I’ve a dataset about 20k images, but very different from the original dataset which approach should I use?</p>