Trained model prediction is not consistent on the same data

rcshubhadeep: <p>Hello All,</p>
<p>I have been trying to classify two sets of images. Which are pretty close in looking to each other. One set of image is image of GUI buttons (mainly desktop) <img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/5283339741875914a2b6b7e309e600337eb837c5.png" alt="30" data-base62-sha1="bLWfx7qeePu2qqBLW1vfr18sbZj" width="88" height="25"> Another set is textboxes (mainly desktop as well) <img src="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e6ae977780ef2f2bb81122577c14fd8ac7cb199.png" alt="29" data-base62-sha1="kjSReHo4F76r0P4Zn3tO0gSE3H3" width="215" height="25"> I have 40 images per class inside my train set and 10 per class in my validation set.</p>
<p>I have taken (almost verbatim) the code from the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">Pytorch tutorial</a> And with slight changes (mainly ornamental) I could make it work. I have added tensorboard support and when I look at my training loss (I am using a resnet50 as a fixed feature extractor and then a FC layer on top with 2 class output) my training loss looks like this</p>
<p>0.6 smoothing - <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b69269ad321742d5266e4e8776fdc5dfc59b9f22.png" data-download-href="https://discuss.pytorch.org/uploads/default/b69269ad321742d5266e4e8776fdc5dfc59b9f22" title="Screenshot from 2021-02-09 11-39-45"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b69269ad321742d5266e4e8776fdc5dfc59b9f22_2_690x323.png" alt="Screenshot from 2021-02-09 11-39-45" data-base62-sha1="q36AqIN0ep4tgFeM2wkEldfeHui" width="690" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b69269ad321742d5266e4e8776fdc5dfc59b9f22_2_690x323.png, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b69269ad321742d5266e4e8776fdc5dfc59b9f22.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b69269ad321742d5266e4e8776fdc5dfc59b9f22.png 2x" data-dominant-color="F7F0EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-09 11-39-45</span><span class="informations">850×398 29.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>0.87 smoothing - <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a161508542ebc89cbcdebb6e8bdbcf19ad8f4a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a161508542ebc89cbcdebb6e8bdbcf19ad8f4a9" title="Screenshot from 2021-02-09 11-40-02"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a161508542ebc89cbcdebb6e8bdbcf19ad8f4a9.png" alt="Screenshot from 2021-02-09 11-40-02" data-base62-sha1="f8tVcKXM9PsndbjSBY2RlPXyY2Z" width="690" height="323" data-dominant-color="F6F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-09 11-40-02</span><span class="informations">854×400 29.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Highest smoothing - <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12c06147e09f120007836d786197d3e83d8b99d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/12c06147e09f120007836d786197d3e83d8b99d2" title="Screenshot from 2021-02-09 11-40-12"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12c06147e09f120007836d786197d3e83d8b99d2_2_690x326.png" alt="Screenshot from 2021-02-09 11-40-12" data-base62-sha1="2FSLxhHOBRZHHNHMzXZT0558oqS" width="690" height="326" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12c06147e09f120007836d786197d3e83d8b99d2_2_690x326.png, https://discuss.pytorch.org/uploads/default/original/3X/1/2/12c06147e09f120007836d786197d3e83d8b99d2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/2/12c06147e09f120007836d786197d3e83d8b99d2.png 2x" data-dominant-color="EFEDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-09 11-40-12</span><span class="informations">851×403 31.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It looks okay to me, At least at the highest smoothing, it seems that the loss is decreasing well.</p>
<p>However, when I evaluate the model, with the eval dataset, each evaluation run on the same set of images spit out different prediction. I almost feel like the model is close to random prediction. I am not sure what I am getting wrong here.</p>
<p>Some important details -</p>
<p>1.&gt; I am resizing the images to a 32x32 square image (even though the original ones has many different dimension. and most of the time rectangular) using a torchvison.transforms.</p>
<p>2.&gt; I am normalizing them using [0.5, 0.5, 0.5], [0.5, 0.5, 0.5]</p>
<p>Please help me debug this</p>

ptrblck: <aside class="quote no-group" data-username="rcshubhadeep" data-post="1" data-topic="111395">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rcshubhadeep/48/12285_2.png" class="avatar"> rcshubhadeep:</div>
<blockquote>
<p>It looks okay to me, At least at the highest smoothing, it seems that the loss is decreasing well.</p>
</blockquote>
</aside>
<p>The loss value itself is quite “high” and depending which criterion you are using your model might not be learning enough. Based on the loss curve it also seems that there is initially some decrease in the loss, which hits a plateau at iteration or epoch 40.</p>
<aside class="quote no-group" data-username="rcshubhadeep" data-post="1" data-topic="111395">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rcshubhadeep/48/12285_2.png" class="avatar"> rcshubhadeep:</div>
<blockquote>
<p>However, when I evaluate the model, with the eval dataset, each evaluation run on the same set of images spit out different prediction. I almost feel like the model is close to random prediction. I am not sure what I am getting wrong here.</p>
</blockquote>
</aside>
<p>Are you using <code>model.eval()</code> during the validation? This would make sure to disable dropout layers and to use the running stats in batchnorm layers instead the batch statistics.</p>

rcshubhadeep: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="111395">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The loss value itself is quite “high” and depending which criterion you are using your model might not be learning enough.</p>
</blockquote>
</aside>
<p>Ok I agree. I wanted to go back to the drawing board and start with a model which I want to intentionally over fit (given, I have only 100 images, 50 per class that should not be a too difficult task, or at least that is what I thought). To prove that it can learn something. So I will detail what is happening here.</p>
<p><strong>Case -1</strong></p>
<p>Model -</p>
<pre><code class="lang-auto">class Reshape(torch.nn.Module):
    def forward(self, x):
        return x.reshape(-1, 64 * 7 * 7)


model_conv = nn.Sequential(nn.Conv2d(3, 8, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(8, 16, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(32, 64, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           Reshape(),
                           nn.Linear(3136, 256), nn.ReLU(),
                           nn.Linear(256, 128), nn.ReLU(),
                           nn.Linear(128, 84), nn.ReLU(),
                           nn.Linear(84, 2)
                          )
</code></pre>
<p>Transforms-</p>
<pre><code class="lang-auto">data_transforms = {
    'train': transforms.Compose([
        transforms.Resize((120, 120)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
    'test': transforms.Compose([
        transforms.Resize((120, 120)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
}
</code></pre>
<p>As you may see, I had not put any Dropout or any standard regularization technique so that I give it all chances to over-fit.</p>
<p>This is the loss curve looks like -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d6608c3c1a33ac63500281d1d2892b83dfcab28.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d6608c3c1a33ac63500281d1d2892b83dfcab28" title="Screenshot from 2021-02-10 13-05-03"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d6608c3c1a33ac63500281d1d2892b83dfcab28.png" alt="Screenshot from 2021-02-10 13-05-03" data-base62-sha1="b2Hn69F1jnBC7uVLZE0QbKug8I0" width="690" height="323" data-dominant-color="F6F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-10 13-05-03</span><span class="informations">850×399 27.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So the loss hits a plateau very soon and never decreases. (The loss is pretty bigger too)</p>
<p>Also, this is the lr - <code>optimizer_conv = optim.SGD(model_conv.parameters(), lr=0.01, momentum=0.9)</code></p>
<p>And a scheduler for lr - <code>exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)</code></p>
<p><strong>Case 2</strong><br>
Same model, just changing the lr</p>
<p><code>optimizer_conv = optim.SGD(model_conv.parameters(), lr=0.001, momentum=0.9)</code></p>
<p>Which keeps it very similar (from the numbers printed it does seem like the val loss and acc both are worse than last time though) -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/498dbc018e4cc1798c3856e6a57705daed09bb46.png" data-download-href="https://discuss.pytorch.org/uploads/default/498dbc018e4cc1798c3856e6a57705daed09bb46" title="Screenshot from 2021-02-10 13-12-17"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/498dbc018e4cc1798c3856e6a57705daed09bb46.png" alt="Screenshot from 2021-02-10 13-12-17" data-base62-sha1="auGwesIUGlfIzAqWBfXnxF8RJS6" width="690" height="327" data-dominant-color="F6F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-10 13-12-17</span><span class="informations">850×404 27.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>** Case 3**</p>
<p>Here I go for a simpler model</p>
<p>Model -</p>
<pre><code class="lang-auto">class Reshape(torch.nn.Module):
    def forward(self, x):
        return x.reshape(-1, 16 * 30 * 30)


model_conv = nn.Sequential(nn.Conv2d(3, 8, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(8, 16, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           Reshape(),
                           nn.Linear(14400, 256), nn.ReLU(),
                           nn.Linear(256, 84), nn.ReLU(),
                           nn.Linear(84, 2)
                          )
</code></pre>
<p>The LR is - <code>optimizer_conv = optim.SGD(model_conv.parameters(), lr=0.001, momentum=0.9)</code></p>
<p>Here I experience something strange. For some runs of the whole code a graph like this will be produced -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/cede52742652fa754320a359d2fca8fb9267a361.png" data-download-href="https://discuss.pytorch.org/uploads/default/cede52742652fa754320a359d2fca8fb9267a361" title="Screenshot from 2021-02-10 13-45-49"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/cede52742652fa754320a359d2fca8fb9267a361.png" alt="Screenshot from 2021-02-10 13-45-49" data-base62-sha1="tw2GgLQ6RH9Dx0r0nojugVRuprP" width="690" height="338" data-dominant-color="F6F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-10 13-45-49</span><span class="informations">858×421 28.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Where the loss decrease is not really seen.</p>
<p>But for some others. The following kind of graph will be produced.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/556b98309de050cbef5f076724ce5618555e5a5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/556b98309de050cbef5f076724ce5618555e5a5e" title="Screenshot from 2021-02-10 13-53-50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/556b98309de050cbef5f076724ce5618555e5a5e_2_690x325.png" alt="Screenshot from 2021-02-10 13-53-50" data-base62-sha1="cbF6w5Wb6sPev85XBT9UWifk7Uy" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/556b98309de050cbef5f076724ce5618555e5a5e_2_690x325.png, https://discuss.pytorch.org/uploads/default/original/3X/5/5/556b98309de050cbef5f076724ce5618555e5a5e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/5/556b98309de050cbef5f076724ce5618555e5a5e.png 2x" data-dominant-color="F6F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-10 13-53-50</span><span class="informations">864×408 26.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Which shows that the loss is decreasing well. And also the validation acc reaches 100% (larger than the test one!)</p>
<p>What is wrong here you think? Why the network before was not able to over-fit the data?</p>
<hr>
<p>Also, an interesting insight is that, on the exact same data-set a HOG-SVM classical model reaches about 95% acc with a bit of effort.</p>
<p>So makes me think that whether I need more data. However, I would have expected using a CNN as a fixed feature extraction should have given me a great result.</p>
<hr>
<p>Another update is with the new transform (120, 120) the restnet fine tuning is actually going much better <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> I guess with a 32 x 32 (the resize transform I had before) was too small to distinguish between two images which is really close to each other from look and feel. This is the loss curve when I train the final layer of resnet18 in this new setting -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d864aec1a298715723f28ef81057e5dab7e9b99.png" data-download-href="https://discuss.pytorch.org/uploads/default/8d864aec1a298715723f28ef81057e5dab7e9b99" title="Screenshot from 2021-02-10 14-40-54"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d864aec1a298715723f28ef81057e5dab7e9b99_2_690x371.png" alt="Screenshot from 2021-02-10 14-40-54" data-base62-sha1="kbZ2JiiH59W4MXeddIO3yXLZhI5" width="690" height="371" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d864aec1a298715723f28ef81057e5dab7e9b99_2_690x371.png, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d864aec1a298715723f28ef81057e5dab7e9b99.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d864aec1a298715723f28ef81057e5dab7e9b99.png 2x" data-dominant-color="F6F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-10 14-40-54</span><span class="informations">843×454 29.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<hr>
<p>Another update, some experiments with lr (0.01) and my first model shows that sometimes The loss does decrease. Sometimes not. That can even happen in two consecutive runs. This does not make sense to me. Why the behavior is not consistent if everything else stays the same? What am I missing here?</p>

ptrblck: <aside class="quote no-group" data-username="rcshubhadeep" data-post="3" data-topic="111395">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rcshubhadeep/48/12285_2.png" class="avatar"> rcshubhadeep:</div>
<blockquote>
<p>Another update, some experiments with lr (0.01) and my first model shows that sometimes The loss does decrease. Sometimes not. That can even happen in two consecutive runs. This does not make sense to me. Why the behavior is not consistent if everything else stays the same? What am I missing here?</p>
</blockquote>
</aside>
<p>This might be expected, if your overall training is unstable and you would have to play around with some hyperparameters. Also, if you don’t seed the code and don’t force it to be deterministic, slight variations are expected. In the best case your training would be stable enough to yield approx. the same final performance. However, as your loss curves show, the training gets often stuck.</p>
<p>The model is generally able to overfit random data as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.manual_seed(2809)

model_conv = nn.Sequential(nn.Conv2d(3, 8, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(8, 16, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(32, 64, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Flatten(),
                           nn.Linear(3136, 256), nn.ReLU(),
                           nn.Linear(256, 128), nn.ReLU(),
                           nn.Linear(128, 84), nn.ReLU(),
                           nn.Linear(84, 2)
                          )

data = torch.randn(10, 3, 120, 120)
target = torch.randint(0, 2, (10,))
optimizer = torch.optim.Adam(model_conv.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

nb_epochs = 200
for epoch in range(nb_epochs):
    optimizer.zero_grad()
    output = model_conv(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print('epoch {}, loss {}'.format(epoch, loss.item()))
</code></pre>

rcshubhadeep: <p>I see, I am gonna manual_seed the model and then I will see how it behaves. However, since I see things are okay(ish) with resnet18/34 I may stick to transfer learning. Specifically given how small the data set actually is.</p>