Transfer learning: reduced performance when updating `out_features` directly

dzy1: <p>Hi, I’m training a model using transfer learning on this dataset: <strong><a href="https://github.com/mrdbourke/pytorch-deep-learning/tree/main/data" rel="noopener nofollow ugc">pizza_steak_sushi</a></strong>, and when trying to update the <code>out_features</code> directly, performance is really reduced. Example below</p>
<p>I am using <a href="https://pytorch.org/vision/0.14/models/generated/torchvision.models.efficientnet_b0.html#efficientnet-b0" rel="noopener nofollow ugc">EfficientNet_B0</a></p>
<pre><code class="lang-auto">weights = torchvision.models.EfficientNet_B0_Weights.DEFAULT
model = torchvision.models.efficientnet_b0(weights=weights)
</code></pre>
<p>The original classifier layer has <code>out_features=1000</code> and I would like  <code>out_features=3</code></p>
<p>This is the output of <code>model.classifier</code>:</p>
<pre><code class="lang-auto">Sequential(
  (0): Dropout(p=0.2, inplace=True)
  (1): Linear(in_features=1280, out_features=1000, bias=True)
)
</code></pre>
<p>Freezing:</p>
<pre><code class="lang-auto">for param in model.features.parameters():
    param.requires_grad = False
</code></pre>
<p>Now updating the <code>out_features</code>, if I try to update it directly like so:</p>
<pre><code class="lang-auto">model.classifier[1].out_features = 3
</code></pre>
<p>compared with:</p>
<pre><code class="lang-auto">model.classifier = torch.nn.Sequential(
           nn.Dropout(p=0.2, inplace=True),
           nn.Linear(in_features=1280, out_features=3, bias=True).to(device))
</code></pre>
<p>Even though they both yield the same result:</p>
<pre><code class="lang-auto">Sequential(
  (0): Dropout(p=0.2, inplace=True)
  (1): Linear(in_features=1280, out_features=3, bias=True)
)
</code></pre>
<p>The former performs poorly (training for 5 epochs):</p>
<pre><code class="lang-auto">Epoch: 1 | train_loss: 6.4254 | train_acc: 0.0234 | test_loss: 6.0019 | test_acc: 0.0000
Epoch: 2 | train_loss: 4.0430 | train_acc: 0.2852 | test_loss: 3.5201 | test_acc: 0.1752
Epoch: 3 | train_loss: 2.6871 | train_acc: 0.2539 | test_loss: 2.4525 | test_acc: 0.2585
Epoch: 4 | train_loss: 1.9191 | train_acc: 0.3359 | test_loss: 2.1126 | test_acc: 0.3002
Epoch: 5 | train_loss: 1.6082 | train_acc: 0.3906 | test_loss: 1.9565 | test_acc: 0.3523
</code></pre>
<p>Compared with the later:</p>
<pre><code class="lang-auto">Epoch: 1 | train_loss: 1.0294 | train_acc: 0.4453 | test_loss: 0.8789 | test_acc: 0.6420
Epoch: 2 | train_loss: 0.9224 | train_acc: 0.6445 | test_loss: 0.6745 | test_acc: 0.9062
Epoch: 3 | train_loss: 0.7471 | train_acc: 0.8516 | test_loss: 0.6538 | test_acc: 0.8655
Epoch: 4 | train_loss: 0.6976 | train_acc: 0.7500 | test_loss: 0.6515 | test_acc: 0.8456
Epoch: 5 | train_loss: 0.6853 | train_acc: 0.7734 | test_loss: 0.6107 | test_acc: 0.8665
</code></pre>
<p>Can anyone explain why this is?</p>
<p>Many thanks</p>

Peter_Vennerstrom: <pre><code class="lang-auto">import torch
import torch.nn as nn

# init layer
layer = nn.Linear(100, 100)

print(layer)
print(layer.weight.shape)

# adjust out_features instance attribute without init
layer.out_features = 10

print(layer)
print(layer.weight.shape)

# layer output is still 100 not 10
x = torch.rand((1,100))
y = layer(x) 
print(y.shape)
</code></pre>
<p>Looking at the source code <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html#Linear" rel="noopener nofollow ugc">here</a>, the out_features attribute appears to only be used during <strong>init</strong> to create weights with the correct shape.  Once created those weights are not affected by subsequent changes to out_features.  The example code above illustrates.</p>
<p>If the goal is to keep a portion of the original weights you’ll need to adjust the weights directly keeping the ones you want.  Otherwise, create a new linear layer with the new output shape.</p>

dzy1: <p>Thanks Peter, that cleared it up for me, much appreciated!</p>