Getting different feature vectors from frozen layers after training

Francisco_Yackel: <p>Hi, I’m currently trying to train a classification model which besides the scores, should also return the image’s feature vector. In order to do this, I used a pre-trained MobileNetV2, with a <strong>new</strong> classification layer at the end (with only the classes I need). The objective is to freeze the feature extraction layers and only train the new classification layer. In order to do this, I implemented the model as follows:</p>
<pre><code class="lang-auto">class CustomClassifier(torch.nn.Module):
    def __init__(self, out_features):
        super(CustomClassifier, self).__init__()
        self.model = models.mobilenet_v2(pretrained=True, progress=False)
        classifier = nn.Sequential(nn.Dropout(p=0.2, inplace=False),
                     nn.Linear(in_features=1280, out_features=out_features),
                     nn.LogSoftmax(dim=1))
        self.model.classifier = classifier
        # freeze features parameters
        for p in self.model.features.parameters(): 
            p.requires_grad = False
           
    def forward(self, x):
        return self.model(x), self.model.features(x)
</code></pre>
<p>Then I fit my model, just like this:</p>
<pre><code class="lang-auto">def fit(epochs, model, opt, loss_fn, train_dl, valid_dl, metric):
    # *** MOVE MODEL TO GPU ***
    model = model.to(device)
    
    # PyTorch LR scheduler performs learning rate annealing to boost training performance (optional)
    lr_sched = optim.lr_scheduler.CosineAnnealingLR(opt, len(train_dl)*epochs, 
                                                    eta_min=opt.defaults['lr']/1e6)
    train_start = datetime.now()
    train_losses = []
    val_losses = []
    train_metrics = []
    val_metrics = []
    
    pbar = tqdm(range(1, epochs+1))
    
    # *** TRAINING LOOP ***
    for epoch in pbar:
        train_loss, train_metric, valid_loss, valid_metric = 0., 0., 0., 0.
        
        # *** TRAIN ***
        model.train()
        for xb, yb in train_dl:                     # iterate every batch
            xb, yb = xb.to(device), yb.to(device)   # move features and target to GPU
            output, _ = model(xb)                      # predict
            loss = loss_fn(output, yb)              # compute loss
            loss.backward()                         # backprop
            opt.step()                              # adjust weights
            opt.zero_grad()
            lr_sched.step()                              
            train_loss += loss.item() * len(xb)
            train_metric += metric(output, yb) * len(xb)
            
        # *** VALIDATION ***
        model.eval()
        for xb, yb in valid_dl:
            xb, yb = xb.to(device), yb.to(device)
            with torch.no_grad(): output, _ = model(xb)
            loss = loss_fn(output, yb)
            valid_loss += loss.item() * len(xb)
            valid_metric += metric(output, yb) * len(xb)
   
    # compute metrics and print
        train_loss /= len(train_dl.dataset)
        train_losses.append(train_loss)
        train_metric /= len(train_dl.dataset)
        train_metrics.append(train_metric)
        valid_loss /= len(valid_dl.dataset)
        val_losses.append(valid_loss)
        valid_metric /= len(valid_dl.dataset)
        val_metrics.append(valid_metric)
        pbar.set_description(
            '[{}] loss: train={:.3f}, val={:.3f} -- metric: train={:.3f}, val={:.3f}'.format(
                epoch, train_loss, valid_loss, train_metric, valid_metric
            )
        )
    train_elapsed = datetime.now() - train_start
    print(f'Total time: {train_elapsed.seconds} seg.')
    return train_losses, train_metrics, val_losses, val_metrics
</code></pre>
<p>I’m using these hyperparameters:</p>
<pre><code class="lang-auto">model = CustomClassifier(len(train_ds.classes))
loss_fn = nn.NLLLoss()
metric = accuracy
opt = optim.Adam(model.model.classifier.parameters(), lr=3e-3)
</code></pre>
<p>Finally I call fit method:</p>
<pre><code class="lang-auto">train_losses, train_metrics, val_losses, val_metrics = fit(3, model, opt, loss_fn, train_dl, valid_dl, metric)
</code></pre>
<p>Each time I train the model, I notice the extracted features are different for the <strong>exact same image</strong>. But this is not supposed to happen as those layers are frozen from the beggining, and the same weights are being used when I instance the model (mobilenet pretrained on ImageNet).</p>
<p>Any ideas for what might be causing this inconsistency?</p>
<p>Thanks!!</p>

soulitzer: <p>Hmm, I’m not able to replicate this. The following prints <code>True</code> for me -</p>
<pre><code class="lang-auto">img = next(iter(test_dl))[0]

fit(1, model, opt, loss_fn, train_dl, valid_dl, metric)
_, feature1 = model(img.cuda())
fit(1, model, opt, loss_fn, train_dl, valid_dl, metric)
_, feature2 = model(img.cuda())

print(torch.allclose(feature1, feature2))
</code></pre>

Francisco_Yackel: <p>Hi <a class="mention" href="/u/soulitzer">@soulitzer</a>, first of all thanks for help me!</p>
<p>Mmmm, I think the correct form to replicate this is trying to train two different model versions. So I train a v1 of my model and deploy it. After some time I train another version with more data and deploy it again, but I would like that the feature vector would be the same for the exact same image. I do this example:</p>
<pre><code class="lang-auto">train_losses, train_metrics, val_losses, val_metrics = fit(3, model, opt, loss_fn, train_dl, valid_dl, metric)

train_losses_1, train_metrics_1, val_losses_1, val_metrics_1 = fit(5, model_1, opt_1, loss_fn_1, train_dl, valid_dl, metric_1)
</code></pre>
<p>And then when I do the check:</p>
<pre><code class="lang-auto">img = next(iter(test_dl))[0]
_, feature1 = model(img)
_, feature2 = model_1(img)
print(torch.allclose(feature1, feature2)) #&lt;--- False 
</code></pre>

Alexey_Demyanchuk: <p>Hi, <a class="mention" href="/u/francisco_yackel">@Francisco_Yackel</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> As far as I understand, the most probable explanation of this difference is <code>BatchNorm</code> layers of your backbone. Even then you make <code>requieres_grad = False</code> on backbone layers, forward pass through the network will update running statistics of <code>BatchNorm</code> layers. So, you have to apply <code>eval()</code> method on the <code>BatchNorm</code> layers as well, this will make sure running statistics will not update during training process. This <a href="https://discuss.pytorch.org/t/how-to-freeze-bn-layers-while-training-the-rest-of-network-mean-and-var-wont-freeze/89736/14">thread</a> could be off great help to you if you want more clarification on the topic.</p>
<p>I would also like to notice,  when your fine-tune pretrained model you will probably have a bit different dataset and perhaps would like to actually update those running statistics, so <code>BatchNorm</code> layers could serve as intended (normalizing intermediate representations) for your data as well. Anyway, you can always try both approaches and see what works best for your task at hand.</p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Francisco_Yackel: <p>Hi <a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a>, you are right!!<br>
Thanks for help me!<br>
Adding this into my fit method between model.train() and the loop:</p>
<pre><code class="lang-auto"># *** TRAIN ***
model.train()
for module in model.modules():
    if isinstance(module, nn.BatchNorm2d):
        module.eval()
for xb, yb in train_dl:                     # iterate every batch
</code></pre>
<p>I can fix my inconsistency.<br>
Thanks again <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Alexey_Demyanchuk: <p>Great <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Nice to hear!</p>