Numerical stability of Normal Torch torch.randn

uumami: <p>So I’ve been testing <code>torch.rand</code> and I am not sure if I am doing something wrong. But the routine seems to suffer from some numerical instability. Here is a notebook for replicating my results in colab: <a href="https://colab.research.google.com/drive/1YfzAoLWpU14BTmrEhmrAD--XeizX5j_e?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a><br>
I am using a T4-GPU</p>
<pre><code class="lang-auto">import torch

h = torch.randn(100000000, dtype=torch.float64, device='cuda')
eps =  torch.finfo(torch.float64).eps
print(f'Eps number',eps)
print(f'Mean {h.mean(dtype=torch.float64)}')
print(f'Std {h.std()}')


</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Eps number 2.220446049250313e-16
Mean 3.946074394742064e-05
Std 0.9999776430693459

</code></pre>
<p>Though the mean is close to zero, it is farway from the reported <code>eps</code>. It’s around a factor of 3 bigger than the eps. I ran some bootstrap simulations and it seems to not be biased. Ran thousand simulations of one million points.</p>
<p>Mean:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/913e1969be4bf97fb2f403b778229b90e86b201f.png" data-download-href="https://discuss.pytorch.org/uploads/default/913e1969be4bf97fb2f403b778229b90e86b201f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/913e1969be4bf97fb2f403b778229b90e86b201f_2_599x500.png" alt="image" data-base62-sha1="kIShnqezs7cgpc2z9Abd5simiBN" width="599" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/913e1969be4bf97fb2f403b778229b90e86b201f_2_599x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/1/913e1969be4bf97fb2f403b778229b90e86b201f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/1/913e1969be4bf97fb2f403b778229b90e86b201f.png 2x" data-dominant-color="B3B7BA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">677×565 17.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Std:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffbbb7f99f27411a2282f0d49fb6fa5ae4e4c089.png" data-download-href="https://discuss.pytorch.org/uploads/default/ffbbb7f99f27411a2282f0d49fb6fa5ae4e4c089" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffbbb7f99f27411a2282f0d49fb6fa5ae4e4c089_2_465x500.png" alt="image" data-base62-sha1="AujWiBiFIVn7XhtC9sZsKbp41Jf" width="465" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffbbb7f99f27411a2282f0d49fb6fa5ae4e4c089_2_465x500.png, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffbbb7f99f27411a2282f0d49fb6fa5ae4e4c089.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffbbb7f99f27411a2282f0d49fb6fa5ae4e4c089.png 2x" data-dominant-color="BDC2C5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">570×612 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I also computed the quantiles, and you can see 99% of the simulations is far from zero at least 3 times the eps value.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f90ed2d7e006bdb071dc43eb12bf5b67adc0608.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f90ed2d7e006bdb071dc43eb12bf5b67adc0608" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f90ed2d7e006bdb071dc43eb12bf5b67adc0608_2_649x500.png" alt="image" data-base62-sha1="mLAq7DOH6yExhQMkMS0kzwtZ812" width="649" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f90ed2d7e006bdb071dc43eb12bf5b67adc0608_2_649x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f90ed2d7e006bdb071dc43eb12bf5b67adc0608.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f90ed2d7e006bdb071dc43eb12bf5b67adc0608.png 2x" data-dominant-color="454545"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">722×556 132 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Uumami!</p>
<aside class="quote no-group quote-modified" data-username="uumami" data-post="1" data-topic="196846" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/uumami/48/50579_2.png" class="avatar"> uumami:</div>
<blockquote>
<pre><code class="lang-auto">h = torch.randn(100000000, dtype=torch.float64, device='cuda')
</code></pre>
<p>…</p>
<pre><code class="lang-auto">Mean 3.946074394742064e-05
</code></pre>
<p>Though the mean is close to zero, it is farway from the reported <code>eps</code>.</p>
</blockquote>
</aside>
<p>This is to be expected</p>
<p>The mean of a bunch of (independent, identically-distributed) random<br>
numbers has standard deviation <code>sigma / sqrt (n)</code>, where <code>sigma</code> is<br>
the standard deviation of the individual samples and <code>n</code> is the number of<br>
samples over which you take the mean.*</p>
<p><code>randn()</code> generates samples that have standard deviation <code>1.0</code> and since<br>
you take the mean of <code>1.e8</code> samples, the standard deviation of  your mean<br>
will be <code>1.e-4</code>.  This means that you don’t expect your mean to be “very<br>
close” to zero, but, rather, “about” <code>1.e-4</code> away from zero.</p>
<p>So the mean you calculate of <code>3.946e-5</code> is a perfectly expected value.</p>
<p>*)  This is discussed many places: One reference would be Wikipedia’s<br>
<a href="https://en.wikipedia.org/wiki/Bienaym%C3%A9%27s_identity" rel="noopener nofollow ugc">Bienaymé’s identity.</a>  (Don’t let the generality of this reference put you off;<br>
the simple version we are using is a well-known result from elementary<br>
statistics.)</p>
<p>Best.</p>
<p>K. Frank</p>