Repository: scipy/scipy

Issue:
scipy.stats.spearmanr return values outside range -1 to 1
While calculating the correlation for two vectors (with some missing values), scipy.stats.spearman returns values greater than 1:

In [12]: ss.spearmanr(eur_af_transformed, data['standardized_se'],nan_policy='omit')[0]
Out[12]: 2.5865646154828417

In [2]: scipy.version.version
Out[2]: '0.17.0'

scipy.stats.mstats.spearmanr also returns invalid values:

In [17]: ss.mstats.spearmanr(eur_af_transformed, data['standardized_se'])
Out[17]: SpearmanrResult(correlation=-2.8721254146781465, pvalue=0.0)



Discussion:
Can you please add a reproducible example

@ev-br 

```python
>>> np.random.seed(2017)
>>> x = np.random.rand(3500)
>>> y = np.random.rand(3500)
>>> x[5] = np.nan
>>> r, p = scipy.stats.spearmanr(x, y, nan_policy='omit')
>>> r
374.21526668172135
>>> p
0.0
>>> scipy.__version__
'0.18.1'
>>> np.__version__
'1.11.3'
```

**EDIT:**

The first time I run the example in a Python interpreter, I get the following warning:

`...\Anaconda2\envs\py35\lib\site-packages\scipy\stats\mstats_basic.py:478: RuntimeWarning: overflow encountered in long_scalars
  denom = n*(n**2 - 1)/6.`

Thanks @WaqAli

This has been fixed:

```
In [12]: r, p = spearmanr(x, y, nan_policy='omit')

In [13]: r
Out[13]: 0.029322382934487745

In [14]: p
Out[14]: 
masked_array(data = 0.0828753538887,
             mask = False,
       fill_value = 1e+20)

In [15]: scipy.__version__
Out[15]: '0.19.0rc1'
```

Having `pvalue` a masked array is probably an artefact, which deserves a separate ticket.
Closing this one as resolved.
Thanks for the quick reply. Do you mind telling me what OS you are using? I've seen discrepancies where the bug only occurs on Windows machines. Likely related to this: http://stackoverflow.com/questions/36278590/numpy-array-dtype-is-coming-as-int32-by-default-in-a-windows-10-64-bit-machine
I'm not convinced this isn't still an issue.  I don't have a Windows installation to test this, but here's what I think is happening in `spearmanr` when using Windows (where the default `int` dtype is 32 bits).

The first few lines of `spearmanr` are:

    (x, y, n) = _chk_size(x, y)
    (x, y) = (x.ravel(), y.ravel())

    m = ma.mask_or(ma.getmask(x), ma.getmask(y))
    n -= m.sum()

The value `n` returned from `_chk_size` (3500 in the example) should be a Python integer, which is effectively arbitrary precision in Python 3, so that is not the problem.  But then we do `n -= m.sum()`.  Python integers are immutable, so this is not really an in-place operation.  That is, it really does `n = n - m.sum()`.  `m.sum()` returns a *numpy* integer, and on Windows, that will be a 32 bit integer.  So the new value of `n` is a numpy integer.  Then, when this line is reached,

    denom = n*(n**2 - 1)/6.

the intermediate expression `n*(n**2 - 1)` overflows.
A fix could be as simple as changing `n -= m.sum()` to

    n -= int(m.sum())

so `n` remains a Python integer and won't overflow when `n*(n**2 - 1)` is computed.

@WarrenWeckesser Easier solution than what I was thinking (I was thinking about how to refactor `denom` out so you wouldn't have to cube anything in an intermediate value).

Do you all have Windows CI tests?
OK, I might try to set up an Anaconda install on a Windows machine to test this. No ETA though, so if someone beats me to it, great. It should be easy enough, as https://github.com/scipy/scipy/blob/master/scipy/stats/mstats_basic.py#L416 is pure python, so one can just toss in a breakpoint.

@WaqAli Unfortunately, no, we don't have any Windows CI at the moment. Setting one up would be great (hint, hint :-)).

No worries about the CI. I know setting one up for windows is a non-trivial task. I didn't realize that the definition is in pure python (I actually couldn't track where the exact definition of the function was, I'd found the function you linked to but it has a different API than shown [here](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html)) so as you say full build probably not needed.

Changing the offending line to `n -= int(m.sum())` made it work for me. If that's all there is to it I'll probably submit a short PR later today (unless someone else here wants to do it).

Also just a heads up I think you're "@"-ing the OP, which I am not :)
Oh, sorry Kevin, I should not be pinging strangers on the internet at 2am local time :-). 

Please do submit a PR, ideally with a test. Now could be great time for windows bug fixes as we're lucky to have the CI by Christoph Gohlke. 
@kbrose wrote:

> I actually couldn't track where the exact definition of the function was, I'd found the function you linked to but it has a different API than shown here

Ah, right.  For anyone else who might want to track this down,  the main definition, with signature

    def spearmanr(a, b=None, axis=0, nan_policy='propagate'):

is in `scipy/stats/stats.py`.  In some cases (including Kevin's example), that function calls the function with signature

    def spearmanr(x, y, use_ties=True):

that is defined in `scipy/stats/mstats_basic.py`.

In my earlier comment, I was referring to the function in `mstats_basic.py`.

Resolving patch:
diff --git a/scipy/stats/mstats_basic.py b/scipy/stats/mstats_basic.py
--- a/scipy/stats/mstats_basic.py
+++ b/scipy/stats/mstats_basic.py
@@ -399,7 +399,6 @@ def pearsonr(x,y):
     # point arithmetic.
     r = min(r, 1.0)
     r = max(r, -1.0)
-    df = n - 2
 
     if r is masked or abs(r) == 1.0:
         prob = 0.
@@ -460,7 +459,10 @@ def spearmanr(x, y, use_ties=True):
     (x, y) = (x.ravel(), y.ravel())
 
     m = ma.mask_or(ma.getmask(x), ma.getmask(y))
-    n -= m.sum()
+    # need int() here, otherwise numpy defaults to 32 bit
+    # integer on all Windows architectures, causing overflow.
+    # int() will keep it infinite precision.
+    n -= int(m.sum())
     if m is not nomask:
         x = ma.array(x, mask=m, copy=True)
         y = ma.array(y, mask=m, copy=True)
@@ -530,7 +532,10 @@ def kendalltau(x, y, use_ties=True, use_missing=False):
     if m is not nomask:
         x = ma.array(x, mask=m, copy=True)
         y = ma.array(y, mask=m, copy=True)
-        n -= m.sum()
+        # need int() here, otherwise numpy defaults to 32 bit
+        # integer on all Windows architectures, causing overflow.
+        # int() will keep it infinite precision.
+        n -= int(m.sum())
 
     if n < 2:
         return KendalltauResult(np.nan, np.nan)


Test patch:
diff --git a/scipy/stats/tests/test_mstats_basic.py b/scipy/stats/tests/test_mstats_basic.py
--- a/scipy/stats/tests/test_mstats_basic.py
+++ b/scipy/stats/tests/test_mstats_basic.py
@@ -201,6 +201,19 @@ def test_spearmanr(self):
               0.0, 0.6, 6.7, 3.8, 1.0, 1.2, 1.4, np.nan]
         (x, y) = (ma.fix_invalid(x), ma.fix_invalid(y))
         assert_almost_equal(mstats.spearmanr(x,y)[0], 0.6887299)
+        # Next test is to make sure calculation uses sufficient precision.
+        # The denominator's value is ~n^3 and used to be represented as an
+        # int. 2000**3 > 2**32 so these arrays would cause overflow on
+        # some machines.
+        x = list(range(2000))
+        y = list(range(2000))
+        y[0], y[9] = y[9], y[0]
+        y[10], y[434] = y[434], y[10]
+        y[435], y[1509] = y[1509], y[435]
+        # rho = 1 - 6 * (2 * (9^2 + 424^2 + 1074^2))/(2000 * (2000^2 - 1))
+        #     = 1 - (1 / 500)
+        #     = 0.998
+        assert_almost_equal(mstats.spearmanr(x,y)[0], 0.998)
 
         # test for namedtuple attributes
         res = mstats.spearmanr(x, y)
@@ -223,6 +236,13 @@ def test_kendalltau(self):
                             25,80,80,80,80,80,80, 0,10,45, np.nan, 0])
         result = mstats.kendalltau(x,y)
         assert_almost_equal(np.asarray(result), [-0.1585188, 0.4128009])
+        # make sure internal variable use correct precision with
+        # larger arrays
+        x = np.arange(2000, dtype=float)
+        x = ma.masked_greater(x, 1995)
+        y = np.arange(2000, dtype=float)
+        y = np.concatenate((y[1000:], y[:1000]))
+        assert_(np.isfinite(mstats.kendalltau(x,y)[1]))
 
         # test for namedtuple attributes
         res = mstats.kendalltau(x, y)
@@ -451,7 +471,7 @@ def test_mode_modifies_input(self):
         im[:, :50] += 1
         cp = im.copy()
         a = mstats.mode(im, None)
-        assert_equal(im, cp)    
+        assert_equal(im, cp)
 
 
 class TestPercentile(TestCase):
diff --git a/scipy/stats/tests/test_stats.py b/scipy/stats/tests/test_stats.py
--- a/scipy/stats/tests/test_stats.py
+++ b/scipy/stats/tests/test_stats.py
@@ -594,6 +594,18 @@ def test_spearmanr():
     res2 = stats.spearmanr(x1[:3], x2[:3], nan_policy='omit')
     assert_equal(res1, res2)
 
+    # Regression test for GitHub issue #6061 - Overflow on Windows
+    x = list(range(2000))
+    y = list(range(2000))
+    y[0], y[9] = y[9], y[0]
+    y[10], y[434] = y[434], y[10]
+    y[435], y[1509] = y[1509], y[435]
+    # rho = 1 - 6 * (2 * (9^2 + 424^2 + 1074^2))/(2000 * (2000^2 - 1))
+    #     = 1 - (1 / 500)
+    #     = 0.998
+    x.append(np.nan)
+    y.append(3.0)
+    assert_almost_equal(stats.spearmanr(x, y, nan_policy='omit')[0], 0.998)
 
 class TestCorrSpearmanrTies(TestCase):
     """Some tests of tie-handling by the spearmanr function."""
@@ -683,6 +695,12 @@ def test_kendalltau():
     assert_equal(np.nan, tau)
     assert_equal(np.nan, p_value)
 
+    # Regression test for GitHub issue #6061 - Overflow on Windows
+    x = np.arange(2000, dtype=float)
+    x = np.ma.masked_greater(x, 1995)
+    y = np.arange(2000, dtype=float)
+    y = np.concatenate((y[1000:], y[:1000]))
+    assert_(np.isfinite(stats.kendalltau(x,y)[1]))
 
 def test_kendalltau_vs_mstats_basic():
     np.random.seed(42)