2 Classes ResNet18 CNN with Half the acc during validation step. What can cause this?

auyer: <p>What could cause a validation step to look so much different from a training step ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7b35b40035cccbc1ddff0becc39870416f441f51.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b35b40035cccbc1ddff0becc39870416f441f51" title="bouncy"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7b35b40035cccbc1ddff0becc39870416f441f51_2_296x500.png" alt="bouncy" data-base62-sha1="hzXMOktSE08yDTDu486NIpjGy6l" width="296" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7b35b40035cccbc1ddff0becc39870416f441f51_2_296x500.png, https://discuss.pytorch.org/uploads/default/original/2X/7/7b35b40035cccbc1ddff0becc39870416f441f51.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/7b35b40035cccbc1ddff0becc39870416f441f51.png 2x" data-dominant-color="F6F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bouncy</span><span class="informations">390×657 30.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The network running is almost the same as <a href="https://github.com/CSAILVision/places365/blob/master/train_placesCNN.py" rel="noopener nofollow ugc">this Places365 one</a>, but with only two classes.<br>
The Validation Dataset is equally divided in the two classes.</p>
<p>It almost looks like the network is ignoring one of the classes during training.</p>
<p>Thanks.</p>

ptrblck: <p>Do you calculate the validation accuracy using the complete validation dataset or are you plotting the accuracy sequentially?</p>
<p>Is the training or validation dataset imbalanced?</p>

auyer: <p>The test dataset is a little imbalanced. 1,700,000 for the first class, and  1,000,000 for the second.</p>
<p>I’m plotting it sequentially, logging each validation batch, but I do calculate the average and print it. It it about 50%.<br>
Plotting part:</p>
<pre><code class="lang-auto">...
from tensorboardX import SummaryWriter
writer = SummaryWriter()

...

        if i % args.print_freq == 0:
            print('Epoch: [{0}][{1}/{2}]\t'
                  'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                  'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
                  'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                  'Prec@1 {top1.val:.3f} ({top1.avg:.3f})'.format(
                   epoch, i, len(train_loader), batch_time=batch_time,
                   data_time=data_time, loss=losses, top1=top1))
        writer.add_scalar('loss', loss.item(), globaliter_train/args.print_freq)
        writer.add_scalar('Accuracy', top1.val, globaliter_train/args.print_freq)
        globaliter_train += 1
</code></pre>

ptrblck: <p>Thanks for the information.<br>
In that case your model might overfit on a class.<br>
To counter it, you could use a weighted criterion or oversample the minority class using <code>WeightedRandomSampler</code>.</p>

auyer: <p>I tried using them both in the past, but either I used them wrong or it just did not work.<br>
I created a balanced version of the training dataset, and will post the results here if it solves my issues.<br>
Thanks</p>

auyer: <p>Balanced datasets have not solved my issue.<br>
This is one epoch of validation, and it looks just like the batches from before.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/569a953c418257985365b230464e909c02b7cb18.png" data-download-href="https://discuss.pytorch.org/uploads/default/569a953c418257985365b230464e909c02b7cb18" title="stillNotWorking"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/569a953c418257985365b230464e909c02b7cb18_2_277x500.png" alt="stillNotWorking" data-base62-sha1="cm8fNQwxRprPTl6d4Mjo2wATxHy" width="277" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/569a953c418257985365b230464e909c02b7cb18_2_277x500.png, https://discuss.pytorch.org/uploads/default/original/2X/5/569a953c418257985365b230464e909c02b7cb18.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/569a953c418257985365b230464e909c02b7cb18.png 2x" data-dominant-color="F4F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">stillNotWorking</span><span class="informations">363×655 27.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What could be the issue ?<br>
For comparison, this is a graph using Transfer Learning from the Places365 dataset:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f6ccbee298018bff154117178d108ec6d4fc01a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/f6ccbee298018bff154117178d108ec6d4fc01a7" title="transfer"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f6ccbee298018bff154117178d108ec6d4fc01a7_2_283x500.png" alt="transfer" data-base62-sha1="zdi7DMn7TzQ3pomKUOHGNbMpcur" width="283" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f6ccbee298018bff154117178d108ec6d4fc01a7_2_283x500.png, https://discuss.pytorch.org/uploads/default/original/2X/f/f6ccbee298018bff154117178d108ec6d4fc01a7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f6ccbee298018bff154117178d108ec6d4fc01a7.png 2x" data-dominant-color="F9F1EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">transfer</span><span class="informations">374×659 32.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This has 8 epochs of training and validation.</p>

ptrblck: <p>Does one batch contain 64 samples?<br>
If so, how do you calculate the validation accuracy?</p>

auyer: <p>I just made a correction in the last response. The blue graph represents on epoch of training, not one batch. In that case, the batch size was 112 images, and it looped for 650 times to cover the entire validation dataset of 73.200 ( I just noticed I had a /10 division when plotting to graph, and that’s why it looks like 64/65 setps).</p>
<p>I also found out that I was plotting the top1 value insted of prec1. I do not think this can be the issue, since the average top1 was calculated to be arrount 50%. Nevertheless I will start a test with prec1.val insted of top1.val in a few minutes.<br>
This is what I am doing:</p>
<pre><code class="lang-auto">        # measure accuracy and record loss
        prec1 = accuracy(output.data, target, topk=(1, ))
        losses.update(loss.item(), input.size(0))
        top1.update(prec1.item(), input.size(0))

        writer.add_scalar('Validation Loss', loss.item(), globaliter_val)
        writer.add_scalar('Validation Accuracy', top1.val, globaliter_val)

... 
def accuracy(output, target, topk=(1,)):
    """Computes the precision@k for the specified values of k"""
    maxk = max(topk)
    batch_size = target.size(0)

    _, pred = output.topk(maxk, 1, True, True)
    pred = pred.t()
    correct = pred.eq(target.view(1, -1).expand_as(pred))

    res = []
    for k in topk:
        correct_k = correct[:k].view(-1).float().sum(0)
        res.append(correct_k.mul_(100.0 / batch_size))
    return res
</code></pre>
<p>Update: The change I made to the plot function did not affect the end result.Same shape, but now going to 650 instead of 65.</p>

auyer: <p>I ran the same model wit the same dataset and same parameters using Keras, and it converged.<br>
I’m not sure what is happening with the pytorch version.</p>