Model predicts correct values when used first time but eventually predictions get worse

Adwaitt: <p>Hello everyone! I have a problem with a resnet50 classifier that I trained, its training loss and validation loss were in order and I can say with high confidence that it isn’t overfitting.</p>
<pre><code class="lang-auto">Epoch : 050, Training: Loss: 0.9120, Accuracy: 100.0000%, 
		Validation : Loss : 0.9948, Accuracy: 92.9577%
</code></pre>
<p>Also, it gave all the correct predictions when I used it for inference for the first time but after that, it started giving me wrong predictions. I’ll admit I forgot the model.eval() at first but the prediction was done under the “with torch.no_grad():” so I don’t think it messed up the gradients. Can someone please tell me what’s happening and how I can fix it?</p>
<p>P.S  — Images are RGB</p>
<p>Code for model loading:</p>
<pre><code class="lang-auto">class ClfNetwork(nn.Module):
    def __init__(self):
        super(ClfNetwork, self).__init__()
        self.model = models.resnet50(pretrained = False)
        for param in self.model.parameters():
            param.requires_grad = False
        self.model.fc = nn.Sequential(nn.Linear(2048, 5),
                               nn.Softmax(dim = 1))
    def forward(self, x):
        y = self.model(x)
        return y
    
clf = ClfNetwork().to(DEVICE)
clf.load_state_dict(torch.load('path_to_model.pth')) # &lt;All keys matched successfully&gt;

</code></pre>
<p>Transforms: (I used albumentations)</p>
<pre><code class="lang-auto">ctfms = A.Compose([
    A.Resize(D, D), # D is 224
    A.Normalize(mean = 0.0, std = 1.0, max_pixel_value = 255.0),
    ToTensorV2()
])
</code></pre>
<p>Prediction Function:</p>
<pre><code class="lang-auto">def predict_class(model, transform, Id):
    test_image = io.imread(f'path_to_image_{Id}.tiff')
    test_image_tensor = transform(image = test_image)["image"]
    test_image_tensor = test_image_tensor.view(1, 3, 224, 224).to(DEVICE) # DEVICE is GPU
    with torch.no_grad():
        model.eval()
        out = model(test_image_tensor)
        ps = torch.exp(out)
        predicted_class = ps.argmax()
    return predicted_class
</code></pre>
<p>Do let me know if there are some other details that I need to share. Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="Adwaitt" data-post="1" data-topic="161201">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adwaitt/48/36971_2.png" class="avatar"> Adwaitt:</div>
<blockquote>
<p>its training loss and validation loss were in order and I can say with high confidence that it isn’t overfitting.</p>
</blockquote>
</aside>
<p>The losses and accuracy shows a larger gap, so I would claim your model is overfitting to the training dataset. Especially a training accuracy of 100% doesn’t sound right compared to the lower validation accuracy.</p>
<aside class="quote no-group" data-username="Adwaitt" data-post="1" data-topic="161201">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adwaitt/48/36971_2.png" class="avatar"> Adwaitt:</div>
<blockquote>
<p>I’ll admit I forgot the model.eval() at first but the prediction was done under the “with torch.no_grad():” so I don’t think it messed up the gradients.</p>
</blockquote>
</aside>
<p>That’s correct. Using <code>torch.no_grad()</code> will make sure that no gradients can be calculated. However, performing the forward passes in <code>model.train()</code> will e.g. update the running stats from batchnorm layers and I would thus consider it as a data leak.</p>

Adwaitt: <p>I see. So there is overfitting. But I still am unable to understand how my model was able to work correctly the very first time and after that did not work at all? And also, what solution do you propose to overcome this problem? Please can you also tell me about the data leak, this is the first time I am encountering this issue.</p>