Repository: scikit-hep/pyhf

Issue:
Numerical unstable conversion of significances to p-values
# Description

The conversion of significances to p-values for the asymptotic methods is done by 1 - cdf

https://github.com/scikit-hep/pyhf/blob/73312338ef563305813198e5abc019de46fbbbc2/src/pyhf/infer/calculators.py#L74

This is numerically unstable, since we potentially want to get a very small number by subtracting almost 1 from 1. At a certain point (above ~8 sigma) it will cancel out to 0.

# Expected Behavior

I should get a non-zero p-value.

# Actual Behavior

I got zero.

# Steps to Reproduce

```python
>>> import pyhf
>>> pyhf.__version__ # using current master
'0.4.3'
>>> pyhf.set_backend("numpy")
>>> model = pyhf.simplemodels.hepdata_like(
...     signal_data=[120.0, 110.0], bkg_data=[50.0, 52.0], bkg_uncerts=[3.0, 7.0]
... )
>>> observations = [51, 48]
>>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
>>> pyhf.infer.hypotest(1.0, data, model)
array([0.])
```

# Checklist

- [X] Run `git fetch` to get the most up to date version of `master`
- [X] Searched through existing Issues to confirm this is not a duplicate issue
- [x] Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue

# Possible fix

For the numpy backend one could use `scipy.stats.norm.sf` which seems to be a more numerically stable implementation. Comparing `1-norm.cdf` with `norm.sf` one can also see how the precision is already degraded before the point where the p-value is cancelled out to completely 0:

```python
>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> from scipy.stats import norm
>>> sigma = np.arange(0, 10, 0.1)
>>> one_minus_cdf = 1 - norm.cdf(sigma)
>>> sf = norm.sf(sigma)
>>> plt.plot(sigma, np.abs(one_minus_cdf - sf) / sf)
>>> plt.yscale("log")
>>> plt.show()
```
![image](https://user-images.githubusercontent.com/3707225/85199285-acd7b600-b2ee-11ea-9e2b-ce09d2b89996.png)

Numerical unstable conversion of significances to p-values
# Description

The conversion of significances to p-values for the asymptotic methods is done by 1 - cdf

https://github.com/scikit-hep/pyhf/blob/73312338ef563305813198e5abc019de46fbbbc2/src/pyhf/infer/calculators.py#L74

This is numerically unstable, since we potentially want to get a very small number by subtracting almost 1 from 1. At a certain point (above ~8 sigma) it will cancel out to 0.

# Expected Behavior

I should get a non-zero p-value.

# Actual Behavior

I got zero.

# Steps to Reproduce

```python
>>> import pyhf
>>> pyhf.__version__ # using current master
'0.4.3'
>>> pyhf.set_backend("numpy")
>>> model = pyhf.simplemodels.hepdata_like(
...     signal_data=[120.0, 110.0], bkg_data=[50.0, 52.0], bkg_uncerts=[3.0, 7.0]
... )
>>> observations = [51, 48]
>>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
>>> pyhf.infer.hypotest(1.0, data, model)
array([0.])
```

# Checklist

- [X] Run `git fetch` to get the most up to date version of `master`
- [X] Searched through existing Issues to confirm this is not a duplicate issue
- [x] Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue

# Possible fix

For the numpy backend one could use `scipy.stats.norm.sf` which seems to be a more numerically stable implementation. Comparing `1-norm.cdf` with `norm.sf` one can also see how the precision is already degraded before the point where the p-value is cancelled out to completely 0:

```python
>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> from scipy.stats import norm
>>> sigma = np.arange(0, 10, 0.1)
>>> one_minus_cdf = 1 - norm.cdf(sigma)
>>> sf = norm.sf(sigma)
>>> plt.plot(sigma, np.abs(one_minus_cdf - sf) / sf)
>>> plt.yscale("log")
>>> plt.show()
```
![image](https://user-images.githubusercontent.com/3707225/85199285-acd7b600-b2ee-11ea-9e2b-ce09d2b89996.png)



Discussion:

Resolving patch:
diff --git a/src/pyhf/infer/calculators.py b/src/pyhf/infer/calculators.py
--- a/src/pyhf/infer/calculators.py
+++ b/src/pyhf/infer/calculators.py
@@ -71,7 +71,8 @@ def pvalue(self, value):
 
         """
         tensorlib, _ = get_backend()
-        return 1 - tensorlib.normal_cdf(value - self.shift)
+        # computing cdf(-x) instead of 1-cdf(x) for right-tail p-value for improved numerical stability
+        return tensorlib.normal_cdf(-(value - self.shift))
 
     def expected_value(self, nsigma):
         """
diff --git a/src/pyhf/tensor/pytorch_backend.py b/src/pyhf/tensor/pytorch_backend.py
--- a/src/pyhf/tensor/pytorch_backend.py
+++ b/src/pyhf/tensor/pytorch_backend.py
@@ -1,7 +1,9 @@
 """PyTorch Tensor Library Module."""
 import torch
 import torch.autograd
+from torch.distributions.utils import broadcast_all
 import logging
+import math
 
 log = logging.getLogger(__name__)
 
@@ -329,8 +331,14 @@ def normal_cdf(self, x, mu=0.0, sigma=1.0):
         Returns:
             PyTorch FloatTensor: The CDF
         """
-        normal = torch.distributions.Normal(mu, sigma)
-        return normal.cdf(x)
+        # the implementation of torch.Normal.cdf uses torch.erf:
+        # 0.5 * (1 + torch.erf((value - self.loc) * self.scale.reciprocal() / math.sqrt(2)))
+        # (see https://github.com/pytorch/pytorch/blob/3bbedb34b9b316729a27e793d94488b574e1577a/torch/distributions/normal.py#L78-L81)
+        # we get a more numerically stable variant for low p-values/high significances using erfc(x) := 1 - erf(x)
+        # since erf(-x) = -erf(x) we can replace
+        # 1 + erf(x) = 1 - erf(-x) = 1 - (1 - erfc(-x)) = erfc(-x)
+        mu, sigma = broadcast_all(mu, sigma)
+        return 0.5 * torch.erfc(-((x - mu) * sigma.reciprocal() / math.sqrt(2)))
 
     def poisson_dist(self, rate):
         r"""


Test patch:
diff --git a/tests/test_infer.py b/tests/test_infer.py
--- a/tests/test_infer.py
+++ b/tests/test_infer.py
@@ -1,6 +1,7 @@
 import pytest
 import pyhf
 import numpy as np
+import scipy.stats
 
 
 @pytest.fixture(scope='module')
@@ -157,3 +158,34 @@ def test_calculator_distributions_without_teststatistic(qtilde):
     )
     with pytest.raises(RuntimeError):
         calc.distributions(1.0)
+
+
+@pytest.mark.parametrize(
+    "nsigma,expected_pval",
+    [
+        # values tabulated using ROOT.RooStats.SignificanceToPValue
+        # they are consistent with relative difference < 1e-14 with scipy.stats.norm.sf
+        (5, 2.866515718791945e-07),
+        (6, 9.865876450377018e-10),
+        (7, 1.279812543885835e-12),
+        (8, 6.220960574271829e-16),
+        (9, 1.1285884059538408e-19),
+    ],
+)
+def test_asymptotic_dist_low_pvalues(backend, nsigma, expected_pval):
+    rtol = 1e-8
+    if backend[0].precision != '64b':
+        rtol = 1e-5
+    dist = pyhf.infer.calculators.AsymptoticTestStatDistribution(0)
+    assert np.isclose(np.array(dist.pvalue(nsigma)), expected_pval, rtol=rtol, atol=0)
+
+
+def test_significance_to_pvalue_roundtrip(backend):
+    rtol = 1e-15
+    if backend[0].precision != '64b':
+        rtol = 1e-6
+    sigma = np.arange(0, 10, 0.1)
+    dist = pyhf.infer.calculators.AsymptoticTestStatDistribution(0)
+    pvalue = dist.pvalue(pyhf.tensorlib.astensor(sigma))
+    back_to_sigma = -scipy.stats.norm.ppf(np.array(pvalue))
+    assert np.allclose(sigma, back_to_sigma, atol=0, rtol=rtol)
diff --git a/tests/test_validation.py b/tests/test_validation.py
--- a/tests/test_validation.py
+++ b/tests/test_validation.py
@@ -189,7 +189,7 @@ def expected_result_1bin_normsys(mu=1.0):
     if mu == 1:
         expected_result = {
             "exp": [
-                7.471684419037561e-10,
+                7.47169462e-10,
                 5.7411551509088054e-08,
                 3.6898088062731205e-06,
                 0.00016965731538267896,
@@ -711,7 +711,7 @@ def validate_hypotest(pdf, data, mu_test, expected_result, tolerance=1e-6):
     [
         (setup_1bin_shapesys(), 1e-6),
         (setup_1bin_lumi(), 4e-6),
-        (setup_1bin_normsys(), 1e-6),
+        (setup_1bin_normsys(), 2e-9),
         (setup_2bin_histosys(), 8e-5),
         (setup_2bin_2channel(), 1e-6),
         (setup_2bin_2channel_couplednorm(), 1e-6),