Pytorch dataloader pixel color histogram yields different results for shuffle=True/False

username: FelGC
cooked: <p>In Pytorch, when analyzing the color histogram e.g. of the MNIST dataset using the pytorch data loader, it yields different results when shuffle is True/False. In fact, setting shuffle==True returns the correct result, whereas shuffle==False does not. Both versions (True/False) are supposed to return the set of data objects (i.e. the entire data set)</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms
from tqdm.notebook import tqdm

transform = transforms.Compose([transforms.ToTensor()])
data_set = torchvision.datasets.MNIST(root="./", train=True, download=True, transform=transform)

dl_random = torch.utils.data.DataLoader(data_set, batch_size=1, shuffle=True)
dl_fixed = torch.utils.data.DataLoader(data_set, batch_size=1, shuffle=False)
</code></pre>
<pre><code class="lang-auto">bins = 255
hist2 = torch.zeros(bins)
for data, labels in tqdm(data_set):
    hist2 += torch.histc(data, bins=bins, min=0, max=1)
  
for name, dl in [("fixed",dl_fixed), ("random",dl_random)]:
  bins = 255
  hist = torch.zeros(bins)
  k = 0
  for data, labels in tqdm(dl):
      hist += torch.histc(data, bins=bins, min=0, max=1)
  print(name, (hist-hist2).abs().sum())
</code></pre>
<p>Running the above codes prints e.g.:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; fixed tensor(0.)
&gt;&gt;&gt; random tensor(40.)
</code></pre>
<p>Why does it not yield the same results (“tensor(0.)”)?</p>

username: ptrblck
cooked: <p>The difference is caused in the large number in <code>hist[0]</code> and <code>hist2[0]</code>.<br>
The default <code>float32</code> <code>dtype</code> can represent integers between <code>0</code> and <code>16777216</code> exactly but starts to round for larger (or smaller negative integers) as:</p>
<ul>
<li>Integers between <code>2**24=16777216</code> and <code>2**25=33554432</code> round to a multiple of 2 (even number)</li>
<li>Integers between <code>2**25</code> and <code>2**26</code> round to a multiple of 4</li>
<li>…</li>
<li>Integers between <code>2**n</code> and <code>2**n+1</code> round to a multiple of <code>2**n-23</code>
</li>
</ul>
<p>as given in <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">this Wikipedia article</a>.</p>
<p>As you can see in your code, <code>hist[0]</code> will contain values <code>&gt;2**25</code> so will necessarily apply rounding during the accumulation.<br>
Since the order of operations is not equal, you can’t expect to see the same results when shuffling is enabled or disabled. Use <code>float64</code> to compare such large numbers instead (or <code>long int</code> etc.).<br>
Initializing <code>hist</code> as:</p>
<pre><code class="lang-python">hist = torch.zeros(bins).double()
</code></pre>
<p>gives a zero difference.</p>

username: FelGC
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="142933">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.zeros(bins).double()</code></p>
</blockquote>
</aside>
<p>Thank you very much, that is exactly what happened! I appreciate the rapid help and answer!</p>