Sanity Check : Validation loss not increasing

username: bing
cooked: <p>Hi,<br>
So I am trying to sanity-check my binary image classification model.<br>
I am training it to overfit on 20 samples, now theoretically training loss should decrease and validation loss should increase. Because the model should not be learning anything but both my train &amp; val loss are decreasing here<br>
Validation accuracy is also following a non-random pattern,<br>
Is my assertion for performance expectation true on overfitting on 20 samples and there is something wrong with my training loop/data loading process?<br>
How can I triage it further?<br>
Below are my metric graphs for the experiment -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85880115f54437752abddd155ec15ae8c0c59254.png" data-download-href="https://discuss.pytorch.org/uploads/default/85880115f54437752abddd155ec15ae8c0c59254" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85880115f54437752abddd155ec15ae8c0c59254_2_690x414.png" alt="image" data-base62-sha1="j3gTgcOauXMFdbpKAhCujkUzrGk" width="690" height="414" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85880115f54437752abddd155ec15ae8c0c59254_2_690x414.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85880115f54437752abddd155ec15ae8c0c59254_2_1035x621.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85880115f54437752abddd155ec15ae8c0c59254_2_1380x828.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2178×1308 159 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: KFrank
cooked: <p>Hi Mr. Bing!</p>
<aside class="quote no-group" data-username="bing" data-post="1" data-topic="158876" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<p>So I am trying to sanity-check my binary image classification model.</p>
</blockquote>
</aside>
<p>Your results do look fishy.  In particular, your validation accuracy is roughly<br>
50%, which usually means that your predictions are essentially random<br>
guesses for binary classification, even though your validation loss is going<br>
down to what appears to be a low level.</p>
<p>Could you show us your loss criterion (which I expect and would hope is<br>
<code>BCEWithLogitsLoss</code>), how you compute your accuracy, and a training<br>
accuracy graph to compare with your validation accuracy?</p>
<p>Is your training data balanced?  That is, roughly what fraction of your<br>
training and validation samples are positive?  Could you give us the<br>
true-positive / false-positive / true-negative / false-negative breakdown<br>
of your training and validation predictions?</p>
<p>Best.</p>
<p>K. Frank</p>

username: bing
cooked: <p>Hi K. Frank,<br>
My loss &amp; optimizer looks like the below -</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model_transfer.parameters(), lr=config.lr)
optimizer.zero_grad()
criterion_transfer = nn.BCEWithLogitsLoss()
</code></pre>
<p>I don’t calculate accuracy generally on training data, should I be calculating it?<br>
Nevertheless, I calculated it for this discussion-<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3c6503304d21d6df5d834971c2e547cd36c8dda.png" data-download-href="https://discuss.pytorch.org/uploads/default/b3c6503304d21d6df5d834971c2e547cd36c8dda" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3c6503304d21d6df5d834971c2e547cd36c8dda_2_690x326.png" alt="image" data-base62-sha1="pEmlMYn9jTjqIO2GTJfiVHCM4gG" width="690" height="326" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3c6503304d21d6df5d834971c2e547cd36c8dda_2_690x326.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3c6503304d21d6df5d834971c2e547cd36c8dda_2_1035x489.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3c6503304d21d6df5d834971c2e547cd36c8dda_2_1380x652.png 2x" data-dominant-color="FAFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2690×1274 257 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Data is not balanced. Also, I am just testing it on 20 samples so that I could figure out any potential issues in my training loop<br>
I am calculating accuracy in a below way -</p>
<pre><code class="lang-auto">for batch_idx, (data, target) in enumerate(loader['train']):
     preds = torch.max(output, dim=1, keepdim=True)[1]
     train_total += data.size(0)
     train_correct += np.sum(np.squeeze(preds.eq(target.data.view_as(pred)).cpu().numpy()))
train_accuracy = 100. * (train_correct/train_total)

</code></pre>
<p>and <code>Val Accuracy</code> in a similar way.<br>
Below is the link for log with tp, tn, fp, fn values, I am sorry if it is not much readable, I’ll make it more human readable if you want-</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.file.io/deleted">
  <header class="source">
      <img class="site-icon">

      <a href="https://www.file.io/deleted" target="_blank" rel="noopener nofollow ugc">file.io</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/361;"><img src="https://www.file.io/images/og-img.png" class="thumbnail" width="" height=""></div>

<h3><a href="https://www.file.io/deleted" target="_blank" rel="noopener nofollow ugc">Deleted</a></h3>

  <p>Sharing files with file.io is convenient, anonymous and secure. Just upload files and share the links via email, SMS, Slack, Discord, etc. Easy-to-use REST API.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: KFrank
cooked: <p>Hi Mr. Bing!</p>
<aside class="quote no-group" data-username="bing" data-post="3" data-topic="158876" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<pre><code class="lang-auto">criterion_transfer = nn.BCEWithLogitsLoss()
</code></pre>
<p>I don’t calculate accuracy generally on training data, should I be calculating it?<br>
Nevertheless, I calculated it for this discussion</p>
</blockquote>
</aside>
<p>This shows that both your training and validation accuracies fluctuate<br>
around 50% while both your training and validation losses become<br>
rather low.  I don’t think (in normal usage) that you can get a loss that<br>
low with <code>BCEWithLogitsLoss</code> when your accuracy is 50%.  So I think<br>
that you’re doing something fishy.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">     preds = torch.max(output, dim=1, keepdim=True)[1]
</code></pre>
</blockquote>
</aside>
<p>This looks very odd.  When using <code>BCEWithLogitsLoss</code> for binary<br>
classification, the output of your network would have a single value<br>
(a <em>logit</em>) for each thing (e.g., batch element) you were making a<br>
prediction for.  In such a case you would not compute the <code>argmax()</code><br>
(<code>max()[1]</code>) of the output of your network to get any sort of useful<br>
predictions.</p>
<p>What is the shape of <code>output</code> and what are its values supposed to mean?</p>
<p>You say that you are working with 20 samples.  Could you post the output<br>
of your network (after training) for those 20 samples, together with the<br>
ground truth?</p>
<aside class="quote no-group">
<blockquote>
<p>Below is the link for log with tp, tn, fp, fn values</p>
</blockquote>
</aside>
<p>I didn’t follow the link to your “tp, tn, fp, fn.”  They’re only four numbers.<br>
You could just post them.</p>
<p>Best.</p>
<p>K. Frank</p>

username: bing
cooked: <p>Hi K. Frank,</p>
<p>Below is the shape of output for batch size-1-</p>
<pre><code class="lang-auto">Output Shape- torch.Size([1, 2])
Output- tensor([[0.2294, 0.6553]], device='cuda:0', grad_fn=&lt;GatherBackward&gt;)
Target Shape- torch.Size([1])
Target- tensor([0], device='cuda:0')
true_positives 0.0
false_negatives 1.0
true_negatives 0.0
false_negative 0.0
</code></pre>
<p>Also, adding the training loop, I am using Timm</p>
<pre><code class="lang-auto">import timm
model_transfer = timm.create_model('convnext_tiny_in22k', pretrained=True,num_classes=2)
</code></pre>
<p>and here is the output for batch size 20 for 20 samples only for 1 epoch-</p>
<pre><code class="lang-auto">Output- tensor([[ 0.2294,  0.6553],
        [-0.0444, -0.0695],
        [ 0.1748,  0.3308],
        [ 0.4883,  0.5134],
        [ 0.1779,  0.2593],
        [-0.0736,  0.2526],
        [ 0.7483,  0.4661],
        [-0.2667,  0.4035],
        [ 0.0905,  0.0604],
        [ 0.2678, -0.2460],
        [ 0.3801,  0.1053],
        [ 0.2109,  0.3041],
        [ 0.2760,  0.2031],
        [ 0.1168,  0.5727],
        [ 0.1044,  0.0052],
        [ 0.2459, -0.1016],
        [ 0.1527,  0.1206],
        [-0.0633, -0.0126],
        [ 0.1624, -0.0535],
        [ 0.0871,  0.1854]], device='cuda:0', grad_fn=&lt;GatherBackward&gt;)
Target- tensor([[0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0]], device='cuda:0')
Epoch: 1 	Training Loss: 0.855001 	Validation Loss: 0.877783 	Training Accuracy: 50.000000 	 Validation Accuracy: 80.000000 	 Train_True Positive- 0.0 Train_False Positive- 200.0 Train_True Negative - 200.0 Train_False Negative - 0.0 Val_True Positive- 0.0 Val_False Positive- 4.0 Val_True Negative - 16.0 Val_False Negative - 0.0
Validation loss decreased (inf --&gt; 0.877783).  Saving model ..
</code></pre>

username: KFrank
cooked: <p>Hi Mr. Bing!</p>
<p>I see problem with your data and results.</p>
<aside class="quote no-group" data-username="bing" data-post="5" data-topic="158876" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<pre><code class="lang-auto">true_positives 0.0
false_negatives 1.0
true_negatives 0.0
false_negative 0.0
</code></pre>
</blockquote>
</aside>
<p>First I assume that “<code>false-negatives 1.0</code>” is a typo and should be<br>
“<code>false-positives 1.0</code>”  This means that this single sample (batch<br>
size of 1) is a <strong>negative</strong> sample, incorrectly classified as positive.</p>
<aside class="quote no-group">
<blockquote>
<p>and here is the output for batch size 20 for 20 samples only for 1 epoch-</p>
<pre><code class="lang-auto">Output- tensor([[ 0.2294,  0.6553],
        [-0.0444, -0.0695],
        [ 0.1748,  0.3308],
        [ 0.4883,  0.5134],
        [ 0.1779,  0.2593],
        [-0.0736,  0.2526],
        [ 0.7483,  0.4661],
        [-0.2667,  0.4035],
        [ 0.0905,  0.0604],
        [ 0.2678, -0.2460],
        [ 0.3801,  0.1053],
        [ 0.2109,  0.3041],
        [ 0.2760,  0.2031],
        [ 0.1168,  0.5727],
        [ 0.1044,  0.0052],
        [ 0.2459, -0.1016],
        [ 0.1527,  0.1206],
        [-0.0633, -0.0126],
        [ 0.1624, -0.0535],
        [ 0.0871,  0.1854]], device='cuda:0', grad_fn=&lt;GatherBackward&gt;)
</code></pre>
</blockquote>
</aside>
<p>If I understand this correctly, it means that 10 of the 20 samples are<br>
predicted to be positive – that is the second of the two numbers is<br>
algebraically larger than the first.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">Target- tensor([[0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0],
        [0]], device='cuda:0')
</code></pre>
</blockquote>
</aside>
<p>This says that all 20 of your samples are labelled as being <strong>negative.</strong></p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">Train_True Positive- 0.0 Train_False Positive- 200.0 Train_True Negative - 200.0 Train_False Negative - 0.0
</code></pre>
</blockquote>
</aside>
<p>Here you have 400 results, which I speculate is 20 times 20, but who knows?</p>
<p>But in any event both “False Positive” and “True Negative” mean a label of<br>
<strong>negative.</strong>  So all of your samples are negative samples (with half being<br>
incorrectly classified as positive, just as in your set of 20 samples, above.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">Val_True Positive- 0.0 Val_False Positive- 4.0 Val_True Negative - 16.0 Val_False Negative - 0.0
</code></pre>
</blockquote>
</aside>
<p>For your validation results, again all of your samples have ground-truth<br>
labels of negative (this time with a quarter of them being incorrectly classified<br>
as positive).</p>
<p>Two conclusions:</p>
<p>All of the samples you’ve showed are negative.  That’s not only an unbalanced<br>
data set, but fully useless.</p>
<p>I’m surprised that your model isn’t learning to always (or almost always)<br>
predict negative, so your doing something wrong in your training.  (If you<br>
run a single batch of 20 through your model once and perform a single<br>
backpropagation / optimization step, your model likely won’t learn much<br>
of anything, including not learning to always predict negative.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: bing
cooked: <p>Hi K. Frank,<br>
I was really high on caffeine, I am sorry I calculated my confusion matrix values incorrectly.<br>
Below are my metric for 2 samples of batch size 1 for 1 epoch.<br>
Also, I am not getting your point of model learning to calculate negative values, none of my target values is negative so isn’t what the actual model is calculating right now should be judged on the basis of closest to the actual value.</p>
<pre><code class="lang-auto">Output-
 tensor([[-0.0642, -0.0855]], grad_fn=&lt;AddmmBackward0&gt;)
Target-
 tensor([0])
Pred-
 tensor([[-0.0642]], grad_fn=&lt;MaxBackward0&gt;)
Output-
 tensor([[ 0.1499, -0.2758]], grad_fn=&lt;AddmmBackward0&gt;)
Target-
 tensor([0])
Pred-
 tensor([[0.1499]], grad_fn=&lt;MaxBackward0&gt;)
Epoch: 1 	Training Loss: 0.884152 	Validation Loss: 0.716224 	 Validation Accuracy: 100.000000 	  True Positive: 0 	 false Positive: 0 	 true negative: 1 	 false negative: 0
Validation loss decreased (inf --&gt; 0.716224).  Saving model ...
</code></pre>