Adding a numerically stable inverse CDF of standard Gaussian

ClaraBing: <p>Hi,</p>
<p>The current icdf in <a href="https://pytorch.org/docs/stable/_modules/torch/distributions/normal.html#Normal" rel="noopener nofollow ugc">torch.distributions.Normal</a> seems to be numerically unstable compared to <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html" rel="noopener nofollow ugc">scipy.stats.norm</a>.</p>
<p>I implemented a torch version by translating the <a href="https://github.com/scipy/scipy/blob/master/scipy/special/cephes/ndtri.c" rel="noopener nofollow ugc">scipy source code</a> to python and it works the same as the scipy version. I wonder if we could integrate the new code to the current torch.distributions.Normal. Thank you!</p>

ptrblck: <p>Could you explain the numerical instability a bit? I.e. where is it observed and how <code>scipy</code> deals with it?<br>
I’m sure a PR would be welcome, if the instability is reproducible or breaks certain use cases.</p>

ClaraBing: <p>Numerical instability issues occur when the input to the inverse CDF is very close to 0 or 1.<br>
<code>scipy</code> handles these cases with approximations and here is the screenshot of the comments in <a href="https://github.com/scipy/scipy/blob/master/scipy/special/cephes/ndtri.c" rel="noopener nofollow ugc">scipy’s source code</a>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e481d69d19f215dced3306dff86c5e2209d4a92.png" data-download-href="https://discuss.pytorch.org/uploads/default/2e481d69d19f215dced3306dff86c5e2209d4a92" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e481d69d19f215dced3306dff86c5e2209d4a92_2_517x283.png" alt="image" data-base62-sha1="6Bqt5bTGPzrfsTqmyJIoKsfSbpo" width="517" height="283" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e481d69d19f215dced3306dff86c5e2209d4a92_2_517x283.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e481d69d19f215dced3306dff86c5e2209d4a92_2_775x424.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e481d69d19f215dced3306dff86c5e2209d4a92.png 2x" data-dominant-color="F6F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">954×524 34.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>One example is when <code>y=1-1e-8</code>:</p>
<ul>
<li><code>torch.distributions.Normal(0, 1).icdf(torch.tensor(y))</code> returns <code>tensor(inf)</code> (because <code>torch.erfinv</code> returns <code>tensor(inf)</code>);</li>
<li><code>scipy.stats.norm.ppf(y)</code> returns <code>5.612001243305505</code>.</li>
</ul>

ptrblck: <p>Thanks for the update!<br>
Would you like to create a feature request on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> and post the explanation there, so that we could discuss it there with the module owners and track it?<br>
Btw. thanks for digging into this. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

googlebot: <p>that’s kinda related to float32 precision, as 1e-8 &lt; 2**-23 = 1.19e-7</p>
<p>and<br>
torch.erfinv(torch.tensor([1-2**-23])) → 3.7439</p>
<p>issue is this:</p>
<blockquote>
<p>np.float32(1)-np.float32(1e-8)<br>
1.0<br>
np.frexp(np.float32(1)-np.float32(1e-8))<br>
(0.5, 1)</p>
</blockquote>

ClaraBing: <p>I see, this is great, thank you <a class="mention" href="/u/googlebot">@googlebot</a> !!<br>
Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I think there’s no need for a feature request since using <code>DoubleTensor</code> fixes the issue.</p>
<p>Thank you both for your help and suggestions~!</p>

ad8e: <p>DoubleTensor does not solve the issue with erfinv; it fails at 1e-18. But <code>torch.special.ndtri</code> works to 1e-40 with float32, and to &lt;1e-100 with float64.</p>