Repository: scipy/scipy

Issue:
BUG:  scipy.stats.multiscale_graphcorr p-values are computed differently from literature and other packages
### Describe your issue.

This bug is as described as here: https://github.com/neurodata/hyppo/issues/124. The issue is reproduced here:

p-values do not appear to be computed correctly. Literature review/review of similar independence testing code seems to suggest most people use the approach suggested in Phipson et al., 2011 described in 6.2 to always include the given ordering as a permutation. Should be updated for all permutation-based approaches to my knowledge.



### Reproducing Code Example

```python
from scipy.stats import multiscale_graphcorr
import numpy as np

X = np.arange(0, 25)
Y = np.arange(0, 25)
stat, pval, _ = multiscale_graphcorr(X, Y, reps=100)
print(pval)
print(stat)
>> .01
>> 1
```

p-value obtained is 1/100.


R example using energy package (contains other nonparametric multivariate independence tests using permutation tests), with energy version 1.7-7 and R version 4.0.2:

```R
require(energy)
X = 0:25; Y = 0:25; result = dcor.test(X, Y, R=100)
print(result$p.value)
>> .0099...
print(result$stat)
>> 1
```

p-value obtained is 1/(100 + 1). We should always use the result from Phipson et. al 2011, rather than only in the case where the p-value would otherwise be 0.



### Error message

```shell
There is no error message for this issue. The proposed change is very small. It would require changing this line (https://github.com/neurodata/hyppo/pull/223):


# calculate p-value and significant permutation map through list
pvalue = (null_dist >= stat).sum() / reps

# correct for a p-value of 0. This is because, with bootstrapping
# permutations, a p-value of 0 is incorrect
if pvalue == 0:
    pvalue = 1 / reps
```

to:

```
pvalue = ((null_dist >= stat).sum() + 1) / (1 + reps)
```



### SciPy/NumPy/Python version information

```
1.7.0 1.21.1 sys.version_info(major=3, minor=8, micro=5, releaselevel='final', serial=0)d
```


Discussion:
Hi @sampan501, thanks for the report. Did you write this R code (without having a look at the sources of GPL codes like this energy package)? Please remove anything that might be GPL otherwise it will be a problem if latter we say there is something to fix.
I think that @ebridge2 might know more about that specific R code since he wrote the original issue in `hyppo`. From the issue it appears that the R code is similar but not exactly the same what is in the `energy`.

I'm not sure if this is relevant, but the specific fix that I mentioned was lifted straight from here: https://github.com/neurodata/hyppo/blob/e69fe12b68fde7a86c06e18912728821db4b4362/hyppo/tools/common.py#L464. This is in the `hyppo` package that I maintain and where I had originally written the source code for `multiscale_graphcorr`.
@tupui I removed the GPL code. I think that the bug still stands as mentioned in Phipson et al., 2011
Agreed on +1 in the denominator. I ran across this issue for [`permutation_test`](https://docs.scipy.org/doc//scipy/reference/generated/scipy.stats.permutation_test.html) shortly before that was finished.

@sampan501 would you submit a PR? If so, I can merge.
@mdhaber Yes I will do that this week
@sampan501 just thought I'd ping you about this so we don't forget. Please mention me when you open one. Thanks!

Resolving patch:
diff --git a/scipy/stats/_stats_py.py b/scipy/stats/_stats_py.py
--- a/scipy/stats/_stats_py.py
+++ b/scipy/stats/_stats_py.py
@@ -5221,12 +5221,7 @@ def _perm_test(x, y, stat, reps=1000, workers=-1, random_state=None):
         null_dist = np.array(list(mapwrapper(parallelp, range(reps))))
 
     # calculate p-value and significant permutation map through list
-    pvalue = (null_dist >= stat).sum() / reps
-
-    # correct for a p-value of 0. This is because, with bootstrapping
-    # permutations, a p-value of 0 is incorrect
-    if pvalue == 0:
-        pvalue = 1 / reps
+    pvalue = (1 + (null_dist >= stat).sum()) / (1 + reps)
 
     return pvalue, null_dist
 


Test patch:
diff --git a/scipy/stats/tests/test_stats.py b/scipy/stats/tests/test_stats.py
--- a/scipy/stats/tests/test_stats.py
+++ b/scipy/stats/tests/test_stats.py
@@ -7265,6 +7265,17 @@ def test_dist_perm(self):
         assert_approx_equal(stat_dist, 0.163, significant=1)
         assert_approx_equal(pvalue_dist, 0.001, significant=1)
 
+    @pytest.mark.slow
+    def test_pvalue_literature(self):
+        np.random.seed(12345678)
+
+        # generate x and y
+        x, y = self._simulations(samps=100, dims=1, sim_type="linear")
+
+        # test stat and pvalue
+        _, pvalue, _ = stats.multiscale_graphcorr(x, y, random_state=1)
+        assert_allclose(pvalue, 1/1001)
+
 
 class TestPageTrendTest:
     # expected statistic and p-values generated using R at