Repository: scipy/scipy

Issue:
error-in-scipy-wilcoxon-signed-rank-test-for-equal-series
Hello
As presented here:
http://stackoverflow.com/questions/38454893/error-in-scipy-wilcoxon-signed-rank-test-for-equal-series?noredirect=1#comment64315624_38454893
(without no answer to disprove it as a mistake from me),
I wish to report that maybe there is a problem with the results of scipy.wilcoxon signed rank test;

import scipy as sp
x1=[29.39958, 29.21756, 29.350915, 29.34911, 29.212635]
sp.wilcoxon(x1,x1,zero_method="wilcox",correction=True)
returns statistic=0.0, pvalue=nan

But with zero_method="pratt", it returns statistic=0.0, pvalue=0.043114446783075355 instead.
I think there is a mistake there. The statistics (for z-score, if I am not mistaken) are the same, but the results are different for the p-value, while they should not, actually.
When both series are identical, there should be no good p-value, but this is confident with alpha=0.05.

Maybe "pratt" zero_method is wrongly implemented. It is said to be more conservative, but instead it has also constantly given better /lower p-values on a large array of test values.

**edit** josef

It looks like there are two issues
- corner case when all differences are zero needs special handling to return nan p-value
- current implementation of Pratt method is wrong if there are "zeros", if (x == y).sum() > 0



Discussion:
R package coin raises an exception if diff is zero. I guess we could set the pvalue to nan to match the "wilcox" case, so users still get the zero for the diff instead of an exception.

(Briefly reading the code, I'm not sure whether `r` is empty in "pratt". )

```
> library(coin)
> zz = c(1,2,3,4.5)
> zz2 = c(1,2,3,4.50000001)
> wilcoxsign_test(zz ~ zz2, zero.method="Pratt")

        Asymptotic Wilcoxon-Signed-Rank Test (zeros handled a la Pratt)

data:  y by x (neg, pos)
Z = -1, p-value = 0.3173
alternative hypothesis: true mu is not equal to 0

> wilcoxsign_test(zz ~ zz, zero.method="Pratt")
Error in wilcoxsign_test.IndependenceProblem(object = <S4 object of class "IndependenceProblem">,  : 
  all pairwise differences equal zero
```

I'm not sure about small differences in the two arrays, the pratt version looks a bit strange.

```
>>> stats.wilcoxon(np.array([1,2,3,4.5]), np.array([1,2,3,4.50000001]), zero_method="wilcox", correction=False)
WilcoxonResult(statistic=0.0, pvalue=0.31731050786291415)
>>> stats.wilcoxon(np.array([1,2,3,4.5]), np.array([1,2,3,4.50000001]), zero_method="pratt", correction=False)
WilcoxonResult(statistic=0.0, pvalue=0.06788915486182899)
>>> stats.wilcoxon(np.array([1,2,3,4.5]), np.array([1,2,3,5.0]), zero_method="pratt", correction=False)
WilcoxonResult(statistic=0.0, pvalue=0.06788915486182899)
>>> import scipy
>>> scipy.__version__
'0.17.0'
```

In R pratt and wilcoxon give the same pvalue in the example (?)

```
> wilcoxsign_test(zz ~ zz2, zero.method="Pratt", distribution="asympt")

        Asymptotic Wilcoxon-Signed-Rank Test (zeros handled a la Pratt)

data:  y by x (neg, pos)
Z = -1, p-value = 0.3173
alternative hypothesis: true mu is not equal to 0
```

(R commands taken from here, found by google)
https://stat.ethz.ch/pipermail/r-help/2011-April/274931.html
I actually am not well versed enough in R to say whether they made a mistake too, but the behavior seemed more regular than scipy's, hence my report. Sorry that I can't help more (except by doing tests, if you suggest these)
Based on a quick look, pratt doesn't have unit tests where some values in x and y are identical.

What I find a bit strange in pratt is that the mean `mn` uses count (i.e. number of observations) and not non-identical count len(d[d!=0]).

Also, checking a few more examples with R, scipy agrees if there are no common values, but not if there are common values between x and y.

(bug is starting to look more likely. However, if all d = x - y are zero, then we should just get nans independent of any calculations.)
That was more or less what I observed;)
I think I found the bug, it took me some time to find a reference with the formulas for the normal approximation for the p-values 

https://github.com/scipy/scipy/blob/master/scipy/stats/morestats.py#L2409
both `mn` and `se` need to be adjusted for the zeros. However, because rank are assigned when zeros are still included, it is not just changing `count`

```
if zero_method == "pratt":
    ...
    countz = len(d == 0)
    mn -= countz * (countz + 1.) * 0.25
    se -= countz * (countz + 1.) * (2. * countz + 1.)
```

where countz is the number of zeros

A PR to verify and change this would be welcome with unit tests against R wilcoxsign_test
(I'm not eager enough to get back on the contributor list with commits to set up scipy for development again : )

The second one is more recent and more general but refers to Cureton 1967 for this case.
The above changes are the "Z" in equation 1 and 2 in Cureton

Cureton, Edward E. 1967. “The Normal Approximation to the Signed-Rank Sampling Distribution When Zero Differences Are Present.” Journal of the American Statistical Association 62 (319): 1068–69. doi:10.2307/2283694.

Hudgens, Michael, and Glen Satten. 2002. “Midrank Unification of Rank Tests for Exact, Tied, and Censored Data.” Journal of Nonparametric Statistics 14 (5): 569–81. doi:10.1080/10485250213905.


BTW: this issue is a duplicate of #6398 which should be closed 
(I guess I didn't see it or didn't pay attention during vacation)
Awesome research;)
Sorry for the duplicate, I must have made a mistake posting.
Any chance this will get fixed sometime soon? Just got bit by this bug.
@josef-pkt could you send me the article by Cureton if you have access? I would like to fix this issue

Resolving patch:
diff --git a/scipy/stats/morestats.py b/scipy/stats/morestats.py
--- a/scipy/stats/morestats.py
+++ b/scipy/stats/morestats.py
@@ -2717,11 +2717,11 @@ def wilcoxon(x, y=None, zero_method="wilcox", correction=False,
     Parameters
     ----------
     x : array_like
-        The first set of measurements.
+        The first set of measurements, must be one-dimensional.
     y : array_like, optional
-        The second set of measurements.  If `y` is not given, then the `x`
-        array is considered to be the differences between the two sets of
-        measurements.
+        The second set of measurements, , must be one-dimensional.
+        If `y` is not given, then the `x` array is considered to be the
+        differences between the two sets of measurements.
     zero_method : {"pratt", "wilcox", "zsplit"}, optional. Default is "wilcox".
         "pratt":
             includes zero-differences in the ranking process,
@@ -2764,9 +2764,10 @@ def wilcoxon(x, y=None, zero_method="wilcox", correction=False,
     the alternative that the it is negative (``alternative == 'less'``),
     or vice versa (``alternative == 'greater.'``).
 
-    The test uses a normal approximation to derive the p-value. A typical rule
-    is to require that n > 20. For smaller n, exact tables can be used to find
-    critical values.
+    The test uses a normal approximation to derive the p-value (if
+    ``zero_method == 'pratt'``, the approximation is adjusted as in [5]_).
+    A typical rule is to require that n > 20 ([2]_, p. 383). For smaller n,
+    exact tables can be used to find critical values.
 
     References
     ----------
@@ -2777,6 +2778,10 @@ def wilcoxon(x, y=None, zero_method="wilcox", correction=False,
        Vol. 54, 1959, pp. 655-667. :doi:`10.1080/01621459.1959.10501526`
     .. [4] Wilcoxon, F., Individual Comparisons by Ranking Methods,
        Biometrics Bulletin, Vol. 1, 1945, pp. 80-83. :doi:`10.2307/3001968`
+    .. [5] Cureton, E.E., The Normal Approximation to the Signed-Rank
+       Sampling Distribution When Zero Differences are Present,
+       Journal of the American Statistical Association, Vol. 62, 1967,
+       pp. 1068-1069. :doi:`10.1080/01621459.1967.10500917`
 
     Examples
     --------
@@ -2823,12 +2828,22 @@ def wilcoxon(x, y=None, zero_method="wilcox", correction=False,
 
     if y is None:
         d = asarray(x)
+        if d.ndim > 1:
+            raise ValueError('Sample x must be one-dimensional.')
     else:
         x, y = map(asarray, (x, y))
+        if x.ndim > 1 or y.ndim > 1:
+            raise ValueError('Samples x and y must be one-dimensional.')
         if len(x) != len(y):
-            raise ValueError('Unequal N in wilcoxon. Aborting.')
+            raise ValueError('The samples x and y must have the same length.')
         d = x - y
 
+    if zero_method in ["wilcox", "pratt"]:
+        n_zero = np.sum(d == 0, axis=0)
+        if n_zero == len(d):
+            raise ValueError("zero_method 'wilcox' and 'pratt' do not work if "
+                             "the x - y is zero for all elements.")
+
     if zero_method == "wilcox":
         # Keep all non-zero differences
         d = compress(np.not_equal(d, 0), d, axis=-1)
@@ -2862,6 +2877,9 @@ def wilcoxon(x, y=None, zero_method="wilcox", correction=False,
 
     if zero_method == "pratt":
         r = r[d != 0]
+        # normal approximation needs to be adjusted, see Cureton (1967)
+        mn -= n_zero * (n_zero + 1.) * 0.25
+        se -= n_zero * (n_zero + 1.) * (2. * n_zero + 1.)
 
     replist, repnum = find_repeats(r)
     if repnum.size != 0:


Test patch:
diff --git a/scipy/stats/tests/test_morestats.py b/scipy/stats/tests/test_morestats.py
--- a/scipy/stats/tests/test_morestats.py
+++ b/scipy/stats/tests/test_morestats.py
@@ -914,6 +914,24 @@ def test_wilcoxon_bad_arg(self):
         assert_raises(ValueError, stats.wilcoxon, [1, 2], [1, 2],
                       alternative="dummy")
 
+    def test_zero_diff(self):
+        x = np.arange(20)
+        # pratt and wilcox do not work if x - y == 0
+        assert_raises(ValueError, stats.wilcoxon, x, x, "wilcox")
+        assert_raises(ValueError, stats.wilcoxon, x, x, "pratt")
+        # ranksum is n*(n+1)/2, split in half if method == "zsplit"
+        assert_equal(stats.wilcoxon(x, x, "zsplit"), (20*21/4, 1.0))
+
+    def test_pratt(self):
+        # regression test for gh-6805: p-value matches value from R package
+        # coin (wilcoxsign_test) reported in the issue
+        x = [1, 2, 3, 4]
+        y = [1, 2, 3, 5]
+        with suppress_warnings() as sup:
+            sup.filter(UserWarning, message="Sample size too small")
+            res = stats.wilcoxon(x, y, zero_method="pratt")
+        assert_allclose(res, (0.0, 0.31731050786291415))
+
     def test_wilcoxon_arg_type(self):
         # Should be able to accept list as arguments.
         # Address issue 6070.
@@ -931,7 +949,7 @@ def test_accuracy_wilcoxon(self):
 
         T, p = stats.wilcoxon(x, y, "pratt")
         assert_allclose(T, 423)
-        assert_allclose(p, 0.00197547303533107)
+        assert_allclose(p, 0.0031724568006762576)
 
         T, p = stats.wilcoxon(x, y, "zsplit")
         assert_allclose(T, 441)