Different outputs when using different batch size (only on cuda)

Stapo: <p>I boiled down my issue to a very simple example. This network produces different values (by a small decimal) based on the batch size. Note that the values remain consistent regardless of batch size when using CPU as the device (also, is it normal that the output between using CPU and cuda is different?).</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

device = torch.device('cuda:0')

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(38, 256), nn.ReLU(),
            nn.Linear(256, 96), nn.ReLU(),
            nn.Linear(96, 64)
        )
    def forward(self, x):
        out = self.fc(x)
        return out.squeeze()

data = []
nr_features = 38
batch_size = 2
for i in range(batch_size):
    datapoint = []
    for j in range(nr_features):
        datapoint.append(0.1)
    data.append(datapoint)

data = torch.tensor(np.array(data)).float()
data = data.to(device)
torch.manual_seed(10)
model = Net()
model.to(device)
model.eval()

with torch.no_grad():
    print(model(data)[0])

</code></pre>
<p>We used <code>batch_size = 2</code> which prints the following:</p>
<pre><code class="lang-auto">tensor([-1.0196e-01, -4.0992e-03, -2.4501e-02,  2.7438e-02, -5.2672e-02,
        -1.3104e-02, -8.8756e-02, -9.2451e-04,  1.2064e-02, -1.5039e-02,
         4.3409e-02,  5.0925e-05, -3.3305e-02,  8.6518e-02,  6.3065e-02,
        -6.5984e-02, -4.8023e-02, -5.3082e-02, -2.2278e-02,  8.6566e-02,
         7.1233e-02, -4.8462e-03,  5.7919e-03,  1.4048e-01,  2.6209e-02,
        -6.4638e-02,  1.5295e-02,  3.4366e-02, -6.0082e-03,  2.2381e-02,
         1.1678e-02, -9.3038e-03,  1.0102e-01,  3.3924e-02,  4.5724e-02,
         8.1887e-02,  4.6533e-03,  1.1872e-01, -1.5417e-02, -4.6537e-02,
         8.4816e-02, -2.0553e-02,  1.8199e-02,  7.9428e-02,  1.6323e-02,
        -1.2300e-02, -3.0991e-02, -1.0930e-02, -1.1830e-01,  1.3081e-01,
        -1.4709e-02, -4.3337e-03,  6.4821e-02,  8.3538e-02,  6.3237e-02,
         2.6764e-02,  1.1271e-02,  1.0993e-02, -2.3339e-02,  7.9234e-02,
        -2.5017e-02, -5.9334e-02,  1.2681e-01,  5.2663e-02], device='cuda:0')
</code></pre>
<p>Now, if we set <code>batch_size = 1024</code>, the printed values are slightly different:</p>
<pre><code class="lang-auto">tensor([-1.0196e-01, -4.0855e-03, -2.4509e-02,  2.7435e-02, -5.2668e-02,
        -1.3112e-02, -8.8748e-02, -9.1917e-04,  1.2068e-02, -1.5040e-02,
         4.3407e-02,  5.9143e-05, -3.3300e-02,  8.6515e-02,  6.3070e-02,
        -6.5981e-02, -4.8025e-02, -5.3084e-02, -2.2279e-02,  8.6570e-02,
         7.1227e-02, -4.8462e-03,  5.7977e-03,  1.4048e-01,  2.6217e-02,
        -6.4637e-02,  1.5298e-02,  3.4358e-02, -6.0073e-03,  2.2382e-02,
         1.1673e-02, -9.3165e-03,  1.0103e-01,  3.3919e-02,  4.5729e-02,
         8.1896e-02,  4.6527e-03,  1.1873e-01, -1.5428e-02, -4.6541e-02,
         8.4820e-02, -2.0548e-02,  1.8202e-02,  7.9432e-02,  1.6322e-02,
        -1.2294e-02, -3.0990e-02, -1.0929e-02, -1.1830e-01,  1.3081e-01,
        -1.4717e-02, -4.3397e-03,  6.4816e-02,  8.3547e-02,  6.3231e-02,
         2.6767e-02,  1.1276e-02,  1.1000e-02, -2.3333e-02,  7.9229e-02,
        -2.5015e-02, -5.9328e-02,  1.2680e-01,  5.2660e-02], device='cuda:0')
</code></pre>
<p>What’s going on here and how can I reach deterministic outputs on cuda regardless of batch size?</p>
<p>Edit: It turns out that it happens with CPU as well, but seems to be more rare.</p>

ptrblck: <aside class="quote no-group" data-username="Stapo" data-post="1" data-topic="155886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stapo/48/20709_2.png" class="avatar"> Stapo:</div>
<blockquote>
<p>how can I reach deterministic outputs on cuda regardless of batch size?</p>
<p>Edit: It turns out that it happens with CPU as well, but seems to be more rare.</p>
</blockquote>
</aside>
<p>There is no guarantee to create bitwise identical results for different workloads as different algorithms can be used internally on the GPU as well as CPU.<br>
If you stick to the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> you can expect to see deterministic results for the same workload as well as framework and library versions, but not for different use cases / workloads.</p>

Stapo: <p>Oh, I see, so it’s expected behaviour. I expected just batch size to not affect that (despite same machine, same environment). Thanks, will read the docs.</p>

Tonis: <p>It’s absolutely not the expected behaviour. This also happened on my RTX A6000 card, but produced consistent results on my RTX 3090.</p>
<p>This is the results on A6000:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ceac6e70b506a90cc4ed70c41f97d77b774b4f36.png" data-download-href="https://discuss.pytorch.org/uploads/default/ceac6e70b506a90cc4ed70c41f97d77b774b4f36" title="WechatIMG682"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ceac6e70b506a90cc4ed70c41f97d77b774b4f36_2_690x252.png" alt="WechatIMG682" data-base62-sha1="tujN4wjZO5ykclwB7hBerfndoou" width="690" height="252" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ceac6e70b506a90cc4ed70c41f97d77b774b4f36_2_690x252.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ceac6e70b506a90cc4ed70c41f97d77b774b4f36_2_1035x378.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ceac6e70b506a90cc4ed70c41f97d77b774b4f36_2_1380x504.png 2x" data-dominant-color="151616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WechatIMG682</span><span class="informations">1508×552 49.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>This is absolutely the expected behavior since there is no guarantee the same algorithms on different devices will be used as already explained.</p>