Why batch size influence the value of loss?

username: jerly-hjzhou
cooked: <p>I’m using resnet-18 to infer test set. When I set the batch_size=10, I got average loss =0.96381098 as in this graph<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5fcc466e1f5334554c356bdd27877e4dd245b8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/c5fcc466e1f5334554c356bdd27877e4dd245b8a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5fcc466e1f5334554c356bdd27877e4dd245b8a_2_566x500.png" alt="image" data-base62-sha1="sftBZYf3Wq2COJmrmA56COh7hBU" width="566" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5fcc466e1f5334554c356bdd27877e4dd245b8a_2_566x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5fcc466e1f5334554c356bdd27877e4dd245b8a_2_849x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5fcc466e1f5334554c356bdd27877e4dd245b8a.png 2x" data-dominant-color="F4F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">931×822 77.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> .<br>
But when I set batch_size=100, I got average loss=0.96381102 as you can see in this graph<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35376fe186940cfd042a25b2e90d9181115360b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/35376fe186940cfd042a25b2e90d9181115360b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/35376fe186940cfd042a25b2e90d9181115360b1_2_690x462.png" alt="image" data-base62-sha1="7AM4UzyWFnuSmdnrnr7CAc8RKrn" width="690" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/35376fe186940cfd042a25b2e90d9181115360b1_2_690x462.png, https://discuss.pytorch.org/uploads/default/original/3X/3/5/35376fe186940cfd042a25b2e90d9181115360b1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/5/35376fe186940cfd042a25b2e90d9181115360b1.png 2x" data-dominant-color="F4F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">921×618 60.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It’s very strange. My code is below.</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets, transforms
import time
import torch.nn.functional as F

def testReability():
    lossSum = 0.0
    lossNumebr = 0
    for i in range(1):
        model = torch.load('resnet_18-cifar_10.pth')
        model.to(device)
        model.eval()
        for data, target in test_loader:
            data = data.to(device)
            target = target.to(device)
            output = model(data)
            lossSum = lossSum + F.cross_entropy(output, target, reduction='sum').item()
        lossNumebr = lossNumebr + len(test_loader) * batchSize
        curLoss = lossSum / lossNumebr
        print("the {} iteration, loss value is {:.8f}".format(i+1, curLoss))

if __name__ == '__main__':
    batchSize=100
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    print("device is ", device)
    data_transforms = transforms.Compose([
            transforms.ToTensor(),
             transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        ])
    image_datasets = datasets.CIFAR10(root='../data', train = False,
                                              download=True, transform=data_transforms)
    test_loader = torch.utils.data.DataLoader(image_datasets, batch_size=batchSize,
                                              shuffle=True, num_workers=0)
    since = time.time()
    testReability()
    time_elapsed = time.time() - since
    print('Test complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
</code></pre>

username: ptrblck
cooked: <p>The difference is <code>0.96381102 - 0.96381098  = 3.9999999978945766e-08</code>, which is most likely caused by the limited floating point precision using <code>float32</code>. Internally the order of the operations might not be the same or different algorithms might be used depending on the batch size and thus workload.</p>