Repository: gammapy/gammapy

Issue:
Cash statistics : Inconsistency between doc/code or pvalue/siginificance
Gammpay 0.17, official

Hello
In the Cash count statistics description in the "Stats-Statistics" docs
https://docs.gammapy.org/0.17/stats/index.html#cash-counts-statistic

The text says that *the significance is the square root of TS ... which is equivalent to the Li&Ma significance for Li&Ma...*

The statement is true but this is not what is in ```stat.signifcance```.

As you know the Li&Ma usual formula gives a significance which is "single-sided", i.e. that corresponds to a ```1-p_value``` obtained by the integration of the normal distribution from ```-Inf``` to the significance value (that can be negative). 

I have checked on the example that from the p_value the significance should be:
```
from scipy.stats import norm, Chi2
p_value = 1 - chi2.cdf(stat.delta_ts) # same value found : 7.365232895800901
sig_LiMa = norm.ppf(1-p_value)
```
that gives ``` 2.475649754526797```

whereas the results computed in the notebook is :```2.71389625737626``` 
I checked that this corresponds to a ``` p_value``` shared on both sides of the normal function:
```norm.ppf(1-p_value/2)```

As a consequence, if I am correct:
- the text is misleading because it mentions Li&Ma, and the situation should be explicit;
- the significance attribute should be documented as the "double-sided" significance.

However I have checked that ```CashCountsStatitics``` can return negative significances which in principle is not possible with a "double-sided" integration. So there might be another inconsistency somewhere....

Thanks!

P.S.: one should never rely on siginiicances, only on p values.



Discussion:
As far as I understand, Li & Ma define their significance as `S = sqrt(-2 log L)`. 
Just before formula (17), they write ` hence, we can directly take the value of (—2 In 2)^1/2 as the significance of the observed result`.
Then they argue that this "significance" is distributed as a chi-square with 1 dof following Wilks theorem. 

This what is done explicitly in the `CountsStatistics`. This sqrt_ts definition of significance yields in practice to a two-sided 'significance' as you correctly noticed. But I think this is exactly what Li& Ma do. 

Of course, significance is frequently defined as 1-sided, which can be confusing. 

A you note the p-value is the better indicator, and this one is correctly computed. In the field people are used to significance or delta TS, so we need to provide these quantities as well.

A better compromise could be to use `sqrt_ts` rather than significance everywhere. No?
@registerrier I'm +1 to use `sqrt_ts` in Gammapy consistently everywhere. I think it's a more precise description of the quantity we actually use...
You are right, the paper is explicit, the significance which is a square root cannot be negative...
I would suggest having a few words in the stat docs to explicit that Li&Ma is double-sided.

What has been misleading in gammapy for me (and made me crazy for a day) is the following:

- The ```p_value``` obtained from ```scipy.stats.norm``` with ```sqrt(delta_TS)``` is half the real ```p_value``` (Li&Ma).
but ```CashCountStatistics``` return the ```p_value``` from ```scipy.stats.chi2```, i.e. ```delta_ts``` (not the sqrt),  and therefore it is correct ! (but not explicit)

- The significance can be negative because it is multiplied by the sign of the excess (that's a bit non-orthodox I think). A negative value is what you expect from a single-sided significance! 

More generally, on top of this, I have recently discovered that  "some" astronomers are used to claim that sqrt(TS) is sigma without noticing that this can be true only if ndof =1. 

Of course, many of these aspects are "negligible" when playing with bright sources but give hard times when you want to make comparisons between 2 analysis of a population...

I think that what should be returned is:
- ```delta_ts``` and 
- the correct ```p_value``` computed from scipy.stats.chi2(delta_ts,ndof).

These are the robust non-ambiguous outputs of the stat. I would not even return sqrt(TS) as the user will take it as THE significance. 
The docs could give the recipes to compute the significance from the p_value, or gammapy.stats can have the generic functions sigma_single_sided(pvalue), sigma_double_sided(pvalue) for the user to get the significance he needs from the p_value...


There are indeed a number of things to be clarified.

In general, things are done in an unorthodox way. 

- Multiplying the Li&Ma significance by the sign of the excess has become a common practice. It implies that one is interested only at positive fluctuations and P(S>s | excess>=0) , i.e. the one sided significance. In this case, it is clear that the `p_value = 0.5 *chi2.isf(ts,1)`, because half of the realizations have been thrown away.

- Similarly in analyses à la Fermi, where the flux normalization is bound to be positive, only the positive excesses are considered  and the TS is no longer distributed as chi2 with 1 d.o.f. In practice the p_value of a given ts is then again `p_value = 0.5 *chi2.sf(ts,1)`. 
This is checked e.g. in Mattox et al (1996) (http://articles.adsabs.harvard.edu/pdf/1996ApJ...461..396M) and visible in this page from the fermion documentation: https://fermipy.readthedocs.io/en/latest/advanced/tsmap.html

So with these conventions, indeed the p_value that we compute for the `ts` in the `CountsStatistic` is incorrect if it is meant as P(TS>ts | excess>=0). 
Similarly, the p_value of the `ts` we provide in a number of estimators where we consider only positive fluctuations should be half of the chi2 survival function.

So as I understand it, we should modify the `CountsStatistic.p_value` to be `0.5 *chi2.sf(ts,1)` and clearly document that we consider only the positive fluctuations and therefore only the upper bound (1 sided p-value).

Another option is not providing the `p_value` computation and leave the user to understand what to do. (I think this is the case in Fermi tools).

What do you think @tstolarczyk  @adonath  @bkhelifi ?


Thanks a lot for these clarifications! And thanks Thierry to have noticed that!
To answer to your question, @registerrier , I would prefer to change the CountsStatistic method.

In parallel, any user wants to know if one excess is not due to background fluctuations with a standard, simple and unambiguous manner. In this respect, the threshold of 5 for the Li&Ma significance is a simple manner. And people wants that, not a ts, or a p-value.... because, except if they are experts, nobody will know if it is one-sided, or two-sided, and then which formula to use. Also, it would imply that according to the user one might have one formula or an other.
I think that we really need a 'significance' variable, computed correctly depending of the case (norm>0, removal of negative excess, etc). This is safer than a sqrt_ts that is always positive...

Anything left to be done here @registerrier and @tstolarczyk ?

Resolving patch:
diff --git a/gammapy/stats/counts_statistic.py b/gammapy/stats/counts_statistic.py
--- a/gammapy/stats/counts_statistic.py
+++ b/gammapy/stats/counts_statistic.py
@@ -18,13 +18,17 @@ def ts(self):
 
     @property
     def sqrt_ts(self):
-        """Return statistical significance of measured excess."""
+        """Return statistical significance of measured excess.
+           The sign of the excess is applied to distinguish positive and negative fluctuations.
+        """
         return np.sign(self.n_sig) * np.sqrt(self.ts)
 
     @property
     def p_value(self):
-        """Return p_value of measured excess."""
-        return chi2.sf(self.ts, 1)
+        """Return p_value of measured excess.
+           Here the value accounts only for the positive excess significance (i.e. one-sided).
+        """
+        return 0.5*chi2.sf(self.ts, 1)
 
     def compute_errn(self, n_sigma=1.0):
         """Compute downward excess uncertainties.


Test patch:
diff --git a/gammapy/stats/tests/test_counts_statistic.py b/gammapy/stats/tests/test_counts_statistic.py
--- a/gammapy/stats/tests/test_counts_statistic.py
+++ b/gammapy/stats/tests/test_counts_statistic.py
@@ -7,12 +7,12 @@
 ref_array = np.ones((3, 2, 4))
 
 values = [
-    (1, 2, [-1.0, -0.78339367, 0.433396]),
-    (5, 1, [4.0, 2.84506224, 4.4402745934e-3]),
-    (10, 5, [5.0, 1.96543726, 0.049363650550]),
-    (100, 23, [77.0, 11.8294207, 2.75033324833345e-32]),
-    (1, 20, [-19, -5.65760863, 1.534966634510499e-08]),
-    (5 * ref_array, 1 * ref_array, [4.0, 2.84506224, 4.4402745934e-3]),
+    (1, 2, [-1.0, -0.78339367, 0.216698]),
+    (5, 1, [4.0, 2.84506224, 2.220137e-3]),
+    (10, 5, [5.0, 1.96543726, 0.024682]),
+    (100, 23, [77.0, 11.8294207, 1.37e-32]),
+    (1, 20, [-19, -5.65760863, 7.5e-09]),
+    (5 * ref_array, 1 * ref_array, [4.0, 2.84506224, 2.220137e-3]),
 ]
 
 
@@ -83,16 +83,16 @@ def test_cash_excess_matching_significance(mu_bkg, significance, result):
 
 
 values = [
-    (1, 2, 1, [-1.0, -0.5829220133009171, 0.55994580085]),
-    (5, 1, 1, [4.0, 1.7061745691234782, 0.087975582112]),
-    (10, 5, 0.3, [8.5, 3.5853812867949024, 3.365860865528742e-4]),
-    (10, 23, 0.1, [7.7, 3.443415522820395, 5.74416016688779e-4]),
-    (1, 20, 1.0, [-19, -4.590373638528086, 4.424532535784618e-06]),
+    (1, 2, 1, [-1.0, -0.5829220133009171, 0.279973]),
+    (5, 1, 1, [4.0, 1.7061745691234782, 0.043988]),
+    (10, 5, 0.3, [8.5, 3.5853812867949024, 1.68293e-4]),
+    (10, 23, 0.1, [7.7, 3.443415522820395, 2.87208e-4]),
+    (1, 20, 1.0, [-19, -4.590373638528086, 2.2122e-06]),
     (
         5 * ref_array,
         1 * ref_array,
         1 * ref_array,
-        [4.0, 1.7061745691234782, 0.087975582112],
+        [4.0, 1.7061745691234782, 0.043988],
     ),
 ]
 
@@ -110,9 +110,9 @@ def test_wstat_basic(n_on, n_off, alpha, result):
 
 
 values = [
-    (5, 1, 1, 3, [1, 0.422261, 0.672834, 0.178305]),
-    (5, 1, 1, 1, [3.0, 1.29828, 0.19419, 1.685535]),
-    (5, 1, 1, 6, [-2, -0.75585, 0.4497382, 0.571311]),
+    (5, 1, 1, 3, [1, 0.422261, 0.336417, 0.178305]),
+    (5, 1, 1, 1, [3.0, 1.29828, 0.097095, 1.685535]),
+    (5, 1, 1, 6, [-2, -0.75585, 0.224869, 0.571311]),
 ]