Repository: scipy/scipy

Issue:
Spearman's rank correlation coefficient slow with nan values and nan policy = omit
Hello,

I have two arrays of sizes around 8000 reals. The second array contains some nan values.
I want to calculate spearmans rank correlation coefficient and would like to ignore pairs containing nans.

According to the documentation 

>  nan_policy ‘omit’ performs the calculations ignoring nan values. 

So I set the nan_policy to omit. After ten minutes of computation I cancelled the computation.
I examined the underlying code and discovered that nan values are assigned the rank of zero.
Shouldnt they be skipped altogether?

When I ignored the nan values myself like this

```
from numpy import ma
x1 = ma.masked_invalid(array1)
y1 = ma.masked_invalid(array2)
m = ma.mask_or(ma.getmask(x1), ma.getmask(y1))
k = ma.array(x1, mask=m, copy=True).compressed()
j = ma.array(y1, mask=m, copy=True) .compressed()
stats.spearmanr(k,j,nan_policy='omit')
```

the computation ended in the matter of seconds. Something similar is done in the scipy itself with the exception of calling the `.compressed()` function.

I am using Python 3.5.2, Numpy 1.11.2 and scipy 0.18.1.
Many thanks,
Jakub



Discussion:
I can confirm that the handling of `nan` in `scipy.stats.spearmanr` is broken.  Here are some simple examples:

Base case, no `nan`s:

```
In [42]: spearmanr([1, 2, 2.5, 2], [1, 3, 2.5, 4])
Out[42]: SpearmanrResult(correlation=0.316227766016838, pvalue=0.683772233983162)
```

The default is `nan_policy='propagate'`, so the return values of the following should be `nan`:

```
In [43]: spearmanr([1, 2, 2.5, 2, np.nan], [1, 3, 2.5, 4, np.nan])
Out[43]: SpearmanrResult(correlation=0.66688592885535025, pvalue=0.21889398131323154)
```

Using `nan_policy='omit'`, the return value of the following should be the same as the base case:

```
In [44]: spearmanr([1, 2, 2.5, 2, np.nan], [1, 3, 2.5, 4, np.nan], nan_policy='omit')
Out[44]: 
SpearmanrResult(correlation=0.34999999999999998, pvalue=masked_array(data = 0.6499999999999996,
             mask = False,
       fill_value = 1e+20)
)
```

Also, `pvalue` in the above result should not be a masked array.  The use of masked arrays in `spearmanr` is an implementation detail.

I guess `rankdata` doesn't "propagate" nans, so we don't get automatic propagation.

@jaksmid  can you time the mstats version without nans to see whether it's the implementation differences between stats and mstats version or if it's really the nan handling.
I don't see an inherent reason why the masked version should be slow.

@WarrenWeckesser  to the last point, In the multivariate case we might have missing values in the final results if for example one column (axis=0) is all nans.
So, if the stats function shouldn't return a masked array, then the results need to be `nan`ed

(aside: I was just looking at spearmanr for another purpose, but without nans)

I dug deeper and I think the explanation for the larger times and for incorrect results reported by @WarrenWeckesser are that the NaN values are not ignored.

Lets say I call `stats.spearmanr(first_input,second_input,nan_policy='omit')`, where the first input array contains nan values.

First lines of stats.spearman are:

```
contains_nan, nan_policy = _contains_nan(a, nan_policy)

    if contains_nan and nan_policy == 'omit':
        a = ma.masked_invalid(a)
        b = ma.masked_invalid(b)
        return mstats_basic.spearmanr(a, b, axis)
```

Therefore, nan values are masked in the first array and the computation goes to `mstats_basic.spearmanr`

There the or mask is computed and applied to both arrays along the lines of `x = ma.array(x, mask=m, copy=True)`

Now comes what I believe is the problem:
 `rankx = rankdata(x)`

This does not omit the invalid values as it should but assign the rank of zero instead.
See ipython entries:

```
In [19]: j = [1,2,3,np.nan,-1,np.nan,3,3]
In [20]: x = ma.masked_invalid(j)
In [21]: k = stats.mstats_basic.rankdata(x)
In [22]: k
Out[22]: array([ 2.,  3.,  5.,  0.,  1.,  0.,  5.,  5.])
```

The ties are computed. Zeroes are counted as ties.

```
In [23]: stats.mstats_basic.count_tied_groups(k)
Out[23]: {2: 1, 3: 1}
```

This will produce the incorrect results as well as can complicate the computation (you cant use the simpler formula without ties etc).

Thoughts?

Any progress on this? I just ran into the inconsistency with `nan`s and the `masked_array` object for pvalue.

Resolving patch:
diff --git a/scipy/stats/stats.py b/scipy/stats/stats.py
--- a/scipy/stats/stats.py
+++ b/scipy/stats/stats.py
@@ -3298,11 +3298,11 @@ def spearmanr(a, b=None, axis=0, nan_policy='propagate'):
             b = ma.masked_invalid(b)
 
             if nan_policy == 'propagate':
-                rho, pval = mstats_basic.spearmanr(a, b, axis)
+                rho, pval = mstats_basic.spearmanr(a, b, use_ties=True)
                 return SpearmanrResult(rho * np.nan, pval * np.nan)
 
             if nan_policy == 'omit':
-                return mstats_basic.spearmanr(a, b, axis)
+                return mstats_basic.spearmanr(a, b, use_ties=True)
 
         br = np.apply_along_axis(rankdata, axisout, b)
     n = a.shape[axisout]


Test patch:
diff --git a/scipy/stats/tests/test_stats.py b/scipy/stats/tests/test_stats.py
--- a/scipy/stats/tests/test_stats.py
+++ b/scipy/stats/tests/test_stats.py
@@ -627,6 +627,20 @@ def test_tie1(self):
         pr = stats.pearsonr(xr, yr)
         assert_almost_equal(sr, pr)
 
+    def test_tie2(self):
+        # Test tie-handling if inputs contain nan's
+        # Data without nan's
+        x1 = [1, 2, 2.5, 2]
+        y1 = [1, 3, 2.5, 4]
+        # Same data with nan's
+        x2 = [1, 2, 2.5, 2, np.nan]
+        y2 = [1, 3, 2.5, 4, np.nan]
+
+        # Results for two data sets should be the same if nan's are ignored
+        sr1 = stats.spearmanr(x1, y1)
+        sr2 = stats.spearmanr(x2, y2, nan_policy='omit')
+        assert_almost_equal(sr1, sr2)
+
 
 #    W.II.E.  Tabulate X against X, using BIG as a case weight.  The values
 #    should appear on the diagonal and the total should be 899999955.