Does feature extraction has an effect on the Learning Rate?

username: alx
cooked: <p>Hi all,</p>
<p>When I’m loading my pretrained model without freezing or removing any layers and using an lr of 0.001 I’m getting quite good results. Ex:</p>
<pre><code class="lang-auto">models.resnet18(pretrained=True)

def get_trainable(model_params):
    return (p for p in model_params if p.requires_grad)

  
optimizer = torch.optim.Adam(
    get_trainable(model.parameters()),
    lr=0.001,
)
</code></pre>
<p>However, when I do freeze layers for feature extraction my doesn’t learn an all. Ex:</p>
<pre><code class="lang-auto">def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False

n_classes = 5
set_parameter_requires_grad(model, True)
num_ftrs = model.classifier.in_features
model.classifier = nn.Linear(num_ftrs, n_classes)
</code></pre>
<p>There’s a third scenario I just ran into where I don’t freeze anything but specify the number of classes. In this case, my model overfits.</p>
<pre><code class="lang-auto">model.classifier = nn.Linear(num_ftrs, n_classes)
</code></pre>
<p>Any idea why this happens?</p>

username: JuanFMontesinos
cooked: <p>Hi, are you passing the whole model to the optimizer in the second case?<br>
You have to.</p>
<p>What’s the diff between 3rd situation and 2nd? you seem to be specifying number of classes in both.</p>

username: alx
cooked: <p>The difference between 2nd and 3rd is that for the 2nd scenario I’m using the set_parameter_requires_grad function.</p>
<p>Regarding optimization, I’m using this for both:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(),lr=0.001)
</code></pre>

username: JuanFMontesinos
cooked: <p>It’s still a bit confusing for me. How are you training the 1st case if you aren’t adding the last layer which maps to the amount of classes of your dataset?</p>
<p>It seems you are loading the network without modifying the final amount of classes. In case you do feature extraction this is ok.</p>
<p>Without modifications you are taking ImageNet weights. Those weights come from a very big dataset with more than 1k classes. That’s why it generalizes very well as feature extractor.</p>
<p>When you freeze the whole model but the final layer, your network is not capable to map those features to your classes.<br>
In the 3rd case it seems your dataset is not big enough and you overfit.</p>
<p>How to solve this?<br>
You can think firsts layers are more general, and the deeper the more specific the learned filters are. You can try to use a smaller LR for the pretrained part, such that those weights aren’t too modified and a normal learning rate for the last FC layer you are adding. You can even freeze first layers as those are truly very general.</p>

username: alx
cooked: <p>Maybe this will help make things clearer:</p>
<p>Scenario 1<br>
Trainable params: <strong>11M</strong><br>
Output layer: <strong>1000 classes</strong><br>
Outcome: <strong>Model learns, doesn’t overfit.</strong></p>
<p>Scenario 2<br>
Trainable params: <strong>5k</strong><br>
Output layer: <strong>5 classes</strong><br>
Outcome: <strong>Model learns, overfits.</strong></p>
<p>Scenario 3<br>
Trainable params: <strong>11m</strong><br>
Output layer: <strong>5 classes</strong><br>
Outcome: <strong>Model doesn’t learn.</strong></p>
<p>I don’t mind the trainable params, the questions is why does the ouput layer influences the learning.</p>