Shuffling dataloader produces wildly different results - Test accuracy issue

username: Lewis_Morris
cooked: <p>When checking the test set accuracy score i’ve noticed a fluctuating accuracy on each run. Currently i’m using a batch size of 8.</p>
<p>How does batch size and shuffling effect the output of the model? It appears it makes all the difference, if I give the model all one class in a batch it doesn’t perform well.</p>
<p>I have a few questions.</p>
<ol>
<li>When the data in the dataloader is shuffled should I experiance a different score each time Its run?</li>
</ol>
<p>I average 91-92%, but the accuracy is different every time.</p>
<ol start="2">
<li>
<p>If I set the dataloaders shuffle to FALSE I get an accuracy score of 50%, a huge reduction from when I shuffle the data. Is this normal?</p>
</li>
<li>
<p>If I manually iterate through each image, convert to tensor, normalize and unsqueeze, I also only get 50% accuracy as when I dont shuffle the dataloader? Is this normal.</p>
</li>
</ol>
<p>I’m sure there is something wrong but can’t for the life of me work it out. Any help appreciated.</p>
<p>EDIT:</p>
<p>I’ve also tried changing the batch size and have realised that this also has an effect on the accuracy.</p>
<p>But I am not backpropogating and have gradients switched off. I thought the way it works is that it predicts on each image in the tensor individually? But I guess this isnt correct?</p>
<p>So how do I infer on a single image if I wanted to?</p>
<p>Or am I completly wrong in my thought pattern?</p>
<p>I’ve run some tests and here are my results, is this expected?#</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1af658579875bee5c6be0ca51bbebc64049c7fb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e1af658579875bee5c6be0ca51bbebc64049c7fb" title="batch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1af658579875bee5c6be0ca51bbebc64049c7fb_2_690x368.jpeg" alt="batch" data-base62-sha1="wcvdpjVFd1GD0ZmEXaA0kthFbYv" width="690" height="368" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1af658579875bee5c6be0ca51bbebc64049c7fb_2_690x368.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1af658579875bee5c6be0ca51bbebc64049c7fb_2_1035x552.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1af658579875bee5c6be0ca51bbebc64049c7fb.jpeg 2x" data-dominant-color="CBBBBA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">batch</span><span class="informations">1080×576 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">
test_location = "/home/lewis/leaf/test/"

#transforms 

new_trans_test = transforms.Compose([((image_width_start,image_height_start)),
                                transforms.ToTensor(),
                                transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

#load dataset

test = torchvision.datasets.ImageFolder(root=test_location,transform=new_trans_test)

#get data loader

test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=True)


val_total_correct = 0
val_total_checked = 0 

for i, data in enumerate(test_loader, 0): 
    #check batch with no gradients
    with torch.no_grad():
        inputs, labels = data
        inputs = inputs.to(device)
        labels = labels.to(device)

        #get outputs
        outputs = net(inputs)
        #calculate loss
        #calculate accuracy
        val_total_correct += torch.sum(torch.argmax(outputs,1) == labels).item()
        val_total_checked += len(labels)

    print(f"\rAccuracy {round(val_total_correct/val_total_checked*100,2)}%  {i+1}/{len(test_loader)}     ", end="",flush=True)
</code></pre>

username: Lewis_Morris
cooked: <p>I’ve solved it I think.</p>
<p>I’m not calling model.eval()</p>

username: Shraban_Chatterjee
cooked: <p>i also have the same problem. Is it solved?</p>