Slightly different results for different batch sizes

username: zhaopku
cooked: <p>I am using a VGG19 network to classify CIFAR10 images, yet I have slightly different results for data of different batch sizes. Code:</p>
<blockquote>
<p>f_image = net.forward(Variable(image, requires_grad=True)).data</p>
<p>f_image2 = net.forward(Variable(image[0].unsqueeze(0), requires_grad=True)).data</p>
<p>f_image3 = net.forward(Variable(image[0].unsqueeze(0), requires_grad=True)).data<br>
print(‘-----------------------------------------’)<br>
print(‘f_image - f_image2 = {}’.format((f_image[0] - f_image2).cpu().data.numpy()))<br>
print(‘f_image - f_image3 = {}’.format((f_image[0] - f_image3).cpu().data.numpy()))<br>
print(‘f_image2 - f_image3 = {}’.format((f_image2 - f_image3).cpu().data.numpy()))<br>
print(‘-----------------------------------------’)</p>
</blockquote>
<p>Specifically, image is of size <code>[batch_size, n_channels, height, width]</code>. And I expect <code>f_image[0].unsqueeze(0)</code> to be the same as <code>f_image2</code> and <code>f_image3</code>. However, they are slightly different:</p>
<blockquote>
<hr>
<p>f_image - f_image2 = [[ 0.0000000e+00  3.7252903e-09 -1.8626451e-09  3.7252903e-09<br>
0.0000000e+00 -3.7252903e-09  3.7252903e-09 -1.8626451e-09<br>
3.7252903e-09 -1.3969839e-09]]<br>
f_image - f_image3 = [[ 0.0000000e+00  3.7252903e-09 -1.8626451e-09  3.7252903e-09<br>
0.0000000e+00 -3.7252903e-09  3.7252903e-09 -1.8626451e-09<br>
3.7252903e-09 -1.3969839e-09]]<br>
f_image2 - f_image3 = [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]<br>
-----------------------------------------]</p>
</blockquote>
<p>The difference is very small, is it a bug or expected behavior of PyTorch?</p>

username: ptrblck
cooked: <p>Values smaller than <code>1e-6</code> are due to the limited floating point precision, so you shouldn’t worry about it.</p>
<p>Unrelated to this, but <code>Variables</code> are deprecated since PyTorch <code>0.4.0</code>, so you can just use <code>tensors</code> now.<br>
Also, I wouldn’t recommend to use <code>.data</code> anymore, as this might have some negative side effects, since Autograd cannot track the changes made to the <code>.data</code> attribute. This is not really important in your current code, but might lead to some issues in the future. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: zhaopku
cooked: <p>But even for the same data sample (image[0] in my case), there are small differences. Does the rounding policy of PyTorch have different behaviors when it comes to different batch sizes?</p>

username: ptrblck
cooked: <p>Yes, even for the same data, you might run into floating point precision limits.<br>
You could try to enable deterministic behavior as described <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">here</a>, but I’m not sure if your results will get more deterministic.</p>

username: zhaopku
cooked: <p>Thanks! Actually I have enabled the deterministic behavior. The difference still exists. The deterministic flags ensure that I have the same difference in different runs, but do not eliminate the differences.</p>

username: zhaopku
cooked: <p>So I guess PyTorch might have some “trade-off” policy for samples in the same batch. If you want maximal precision for the 1st sample in a batch, you might lose more precision in other samples of the batch. And the “trade-off” policy is why I slightly different results for different batch sizes. Is my understanding correct?</p>

username: fanlu
cooked: <p>Could you help me to check my problem, <a href="/t/different-batch-sizes-give-different-test-accuracies/24577">Different batch sizes give different test accuracies</a>, Thanks</p>

username: water
cooked: <p>hello,do you solve this problem? For different results, I am very affected in semantic segmentation, I got a very different result for different batch sizes during prediction</p>