Batch size changes (Linear) layer output values

username: Norix
cooked: <p>I’ve noticed that one of my model changes the output/prediction if I change the batch size (number of sequences). After some tests, I’ve found out that <code>nn.Linear</code> layers give slightly different results for the same input values if the batch size is changed.</p>
<p>Running the following code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

import numpy as np
import random as rnd

seed = 101

rnd.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.use_deterministic_algorithms(True)

torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

f = nn.Linear(1024, 64).cuda()
x = torch.ones(16, 300, 1024).float().cuda()

for n in range(x.shape[0]):
    y = f(x)[:n+1]
    z = f(x[:n+1])

    print(torch.equal(y, z), torch.abs(y - z).mean().item())
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">False 2.0721927285194397e-08
False 1.862645149230957e-08
False 1.862645149230957e-08
False 1.862645149230957e-08
False 1.862645149230957e-08
False 1.862645149230957e-08
False 1.5133991837501526e-08
False 1.5133991837501526e-08
False 1.5133991837501526e-08
False 1.5133991837501526e-08
False 1.5133991837501526e-08
False 1.5133991837501526e-08
True 0.0
True 0.0
True 0.0
True 0.0
</code></pre>
<p>The same is also true for <code>nn.Conv1d</code> layers, though seemingly not for <code>nn.Conv2d</code> layers. Setting <code>torch.backends.cudnn.enabled = False</code> seemingly fixes the issue for <code>nn.Conv1d</code>, but not for <code>nn.Linear</code> layers. Finally, this effect only appears on GPU and not if those calculations are run on CPU.</p>
<p>Is this expected behavior (due to some float precision issues) or a bug?<br>
I’m running <code>pytorch 1.9.0 (py3.8_cuda11.1_cudnn8_0)</code> if that makes a difference.</p>

username: eqy
cooked: <p>This would be expected as different batch-sizes will likely dispatch to different algorithms that can have different numerical behavior (e.g., due to differences in how reductions are ordered).</p>
<p>I observe similar behavior with float32 on <code>sm_86</code>, but e.g., changing the workspace size via <code>CUBLAS_WORKSPACE_CONFIG=:16:8</code> seems to reduce the number of batch sizes that produce slightly different results, and using float64 results in equal comparisons throughout. In general however, there aren’t any guarantees for identical behavior for different batch sizes—deterministic is only intended to guarantee results for the same batch size.</p>

username: Norix
cooked: <p>Thank you for the quick answer!</p>
<p>Setting <code>CUBLAS_WORKSPACE_CONFIG</code> to <code>:16:8</code> (instead of my previous <code>:4096:8</code>) indeed reduced the issue quite a lot (at least for <code>nn.Linear</code>). Going from float32 to float64 reduced the numerical differences by several orders of magnitude, but did not eliminate them; at least on my hardware (RTX 3060).</p>
<p>As far as I understood, I would need to always enforce the same batch size (same as used during training), even when predicting only a single sample (i.e. add dummy samples). Otherwise I risk that:<br>
a) I get worse performance, because the model makes different calculations as it did during training.<br>
b) I get different results for the same sample, depending on how many other samples I predict in the same batch.</p>

username: ptrblck
cooked: <p>No, usually you wouldn’t expect a) and b), since the limited numerical precision is expected and neither value is the “true” value as they depend which values are representable in <code>float32</code>.<br>
You can read more about it in <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">this Wikipedia article</a>.<br>
If your model is sensitive to the expected numerical noise, you might need to use a wider <code>dtype</code>, such as <code>float64</code>.</p>