Repository: pandas-dev/pandas

Issue:
Numpy 1.8 `DeprecationWarning` in compat/scipy.py
Not sure how pressing this is, but with `DeprecationWarning` enabled, I notice that numpy 1.8 is raising a warning during the following call to `describe()`.  [side note: enabled DeprecationWarning in my test suite after learning that it was changed in py2.7 to "ignore" by default.]

```
import pandas as pd
import warnings
warnings.simplefilter("once", DeprecationWarning)

df = pd.DataFrame({"A": [1, 2, 3], "B": [1.2, 4.2, 5.2]})
print df.groupby('A')['B'].describe()
```

stdout:

```
$ python test_fail.py 
.../pandas/compat/scipy.py:68: DeprecationWarning: using a non-integer
number instead of an integer will result in an error in the future
  score = values[idx]
```

Here's the full traceback with DeprecationWarning escalated to an error (`warnings.simplefilter("error", DeprecationWarning)`):

```
Traceback (most recent call last):
  File "test_fail.py", line 6, in <module>
    print df.groupby('A')['B'].describe()
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/core/groupby.py", line 343, in wrapper
    return self.apply(curried)
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/core/groupby.py", line 424, in apply
    return self._python_apply_general(f)
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/core/groupby.py", line 427, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, self.obj, self.axis)
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/core/groupby.py", line 883, in apply
    res = f(group)
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/core/groupby.py", line 422, in f
    return func(g, *args, **kwargs)
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/core/groupby.py", line 329, in curried
    return f(x, *args, **kwargs)
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/core/series.py", line 1386, in describe
    lb), self.median(), self.quantile(ub),
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/core/series.py", line 1316, in quantile
    result = _quantile(valid_values, q * 100)
  File "/home/gmd/ENV/pandas-master-2/lib/python2.7/site-packages/pandas-0.13.0_29_g97860a1-py2.7-linux-i686.egg/pandas/compat/scipy.py", line 68, in scoreatpercentile
    score = values[idx]
IndexError: cannot convert index to integer
```



Discussion:
can you show what `values` and `idx` are at that point (and what `valid_values` and `q` are coming in?

```
values == array([ 1.2])
idx == 0.0
valid_values == array([ 1.2])
q == 0.25
```

Can repro this with plain numpy as:

```
In [1]: import warnings
In [2]: warnings.simplefilter('error', DeprecationWarning)
In [3]: np.array(range(10))[1.0]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-3-2e544ff0834f> in <module>()
----> 1 np.array(range(10))[1.0]

IndexError: cannot convert index to integer
In [4]: warnings.simplefilter('ignore', DeprecationWarning)
In [5]: np.array(range(10))[1.0]
Out[5]: 1
```

ahh...i see you are trying to index with floats as the indexer (when its convertible to an int)....its 'accepted'...but in general not a good idea.

I am going to warn on this in 0.14 too, see here:http://pandas.pydata.org/pandas-docs/dev/whatsnew.html#float64index-api-change (the very last part).

I think pandas is generating the float index internally in compat/scipy.py.  If I call `scipy.stats.scoreatpercentile(np.array([ 1.2]), 25.0)` directly with scipy version 0.13.2, I don't see this warning..

if you have your above example (the top example) fail, can you print out those values (they should be the `valid_values`), prob a float. not sure why that would cause a problem (its a numpy array)

Here are the values again for the original example:
`values == array([ 1.2])`, `idx == 0.0`, `valid_values == array([ 1.2])`, `q == 0.25`

Looks to me like pandas has an [old version of scipy `scoreatpercentile`](https://github.com/pydata/pandas/blob/master/pandas/compat/scipy.py#L9) and needs to copy a [new version](https://github.com/scipy/scipy/blob/master/scipy/stats/stats.py#L1369) to prevent indexing a numpy ndarray with a float, which was deprecated in numpy 1.8.

idx is computed by scipy
afaik pandas is just passing simple stuff
can u see where scipy deprecated this (eg the original issue)
maybe need to call a different routine?

I didn't have scipy installed when I the fail, so I think that `idx` is
computed by a copy of a scipy function that lives in pandas.  According to the
compat/scipy.py header, it was copied to avoid a dependency on scipy.

Looks like scipy.stats.scoreatpercentile was last changed in commit
https://github.com/jjhelmus/scipy/commit/1cdd08b62f52fdacb520b18910b8e1a71017ac9c
to accept sequences of percentiles, and also not index an ndarray
with a float..

for reference:
pandas/compat/scipy.py created by https://github.com/pydata/pandas/issues/1092
non-integer ndarray indexing deprecated in https://github.com/numpy/numpy/pull/3243

ahh...I see now...

ok...so basically that module then needs updating....

care to submit a PR (and prob need some tests their)

we have a 'soft' dep on scipy...but this is such a common thing its fine to have it 'built' in

so will call this a 'bug' then

Sure, I will draft a pr
On Jan 2, 2014 5:55 PM, "jreback" notifications@github.com wrote:

> ahh...I see now...
> 
> ok...so basically that module then needs updating....
> 
> care to submit a PR (and prob need some tests their)
> 
> we have a 'soft' dep on scipy...but this is such a common thing its fine
> to have it 'built' in
> 
> so will call this a 'bug' then
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/5824#issuecomment-31492223
> .

note that (still unreleased) numpy 1.9 will have a percentile that should be able to replace scipy.scoreatpercentile both in performance and features.
it uses partition instead of sort which is faster and will support extended axes (ext axes is not merged yet but should be soon)
as you are a user of percentile maybe you want to give numpy 1.9.dev a try and see if it works for you.

@gdraps could you submit a PR for this?

(and for numpy 1.9 should take advantage of the changes)....

@gdraps PR for this?

Sent PR #6740 to fix the core issue, though it doesn't take advantage of numpy.percentile, which has been in numpy in a form that appears compatible with pandas's usage since 1.5, best I can tell.

When I tried to simply replace `scoreatpercentile` in core/frame.py and core/series.py with `numpy.percentile`, while using numpy 1.8, two tests below failed.

```
======================================================================
FAIL: test_timedelta_ops (pandas.tseries.tests.test_timedeltas.TestTimedeltas)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmd/github_shadow/pandas/pandas/tseries/tests/test_timedeltas.py",
  line 203, in test_timedelta_ops
    tm.assert_almost_equal(result, expected)
  File "testing.pyx", line 58, in pandas._testing.assert_almost_equal
  (pandas/srcAssertionError: numpy.timedelta64(2599999999,'ns') !=
  numpy.timedelta64(2600000000,'ns')

======================================================================
FAIL: test_quantile (pandas.tests.test_series.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmd/github_shadow/pandas/pandas/tests/test_series.py", line 2115,
  in test_quantile
    self.assertEqual(q, scoreatpercentile(self.ts.valid(), 10))
AssertionError: -1.2926251727455667 != -1.2926251727455669
----------------------------------------------------------------------
```

the way the fractions are computed are not the same in your function and numpy so you get slight rounding errors, numpy computes:

```
(1 - fraction) * low + fraction * high 
```

while your code has one operation less:

```
low + (high - low) * fraction
```

maybe we could change numpy to that method if you expect it causes issues, but relying on exact results for floating point operations is usually not a good idea in high level programs without tight control on the operations and rounding modes

@gdraps I would be happy just dropping entirely `pandas.compat.scipy/_quantile` in favor of using the numpy method. Then just change the `test_quantile` test to compare against the numpy method iteself. (and just remove this part of the scipy dep). 

Not sure why this was not done originally. Pls also add a test for using `datetime64[ns]` as I suspect this fails. (look at the `isin` method to see how to do this).
.


Resolving patch:
diff --git a/pandas/compat/scipy.py b/pandas/compat/scipy.py
--- a/pandas/compat/scipy.py
+++ b/pandas/compat/scipy.py
@@ -6,88 +6,6 @@
 import numpy as np
 
 
-def scoreatpercentile(a, per, limit=(), interpolation_method='fraction'):
-    """Calculate the score at the given `per` percentile of the sequence `a`.
-
-    For example, the score at `per=50` is the median. If the desired quantile
-    lies between two data points, we interpolate between them, according to
-    the value of `interpolation`. If the parameter `limit` is provided, it
-    should be a tuple (lower, upper) of two values. Values of `a` outside
-    this (closed) interval will be ignored.
-
-    The `interpolation_method` parameter supports three values, namely
-    `fraction` (default), `lower` and `higher`. Interpolation is done only,
-    if the desired quantile lies between two data points `i` and `j`. For
-    `fraction`, the result is an interpolated value between `i` and `j`;
-    for `lower`, the result is `i`, for `higher` the result is `j`.
-
-    Parameters
-    ----------
-    a : ndarray
-        Values from which to extract score.
-    per : scalar
-        Percentile at which to extract score.
-    limit : tuple, optional
-        Tuple of two scalars, the lower and upper limits within which to
-        compute the percentile.
-    interpolation_method : {'fraction', 'lower', 'higher'}, optional
-        This optional parameter specifies the interpolation method to use,
-        when the desired quantile lies between two data points `i` and `j`:
-
-        - fraction: `i + (j - i)*fraction`, where `fraction` is the
-                    fractional part of the index surrounded by `i` and `j`.
-        - lower: `i`.
-        - higher: `j`.
-
-    Returns
-    -------
-    score : float
-        Score at percentile.
-
-    See Also
-    --------
-    percentileofscore
-
-    Examples
-    --------
-    >>> from scipy import stats
-    >>> a = np.arange(100)
-    >>> stats.scoreatpercentile(a, 50)
-    49.5
-
-    """
-    # TODO: this should be a simple wrapper around a well-written quantile
-    # function.  GNU R provides 9 quantile algorithms (!), with differing
-    # behaviour at, for example, discontinuities.
-    values = np.sort(a, axis=0)
-    if limit:
-        values = values[(limit[0] <= values) & (values <= limit[1])]
-
-    idx = per / 100. * (values.shape[0] - 1)
-    if idx % 1 == 0:
-        score = values[idx]
-    else:
-        if interpolation_method == 'fraction':
-            score = _interpolate(values[int(idx)], values[int(idx) + 1],
-                                 idx % 1)
-        elif interpolation_method == 'lower':
-            score = values[np.floor(idx)]
-        elif interpolation_method == 'higher':
-            score = values[np.ceil(idx)]
-        else:
-            raise ValueError("interpolation_method can only be 'fraction', "
-                             "'lower' or 'higher'")
-
-    return score
-
-
-def _interpolate(a, b, fraction):
-    """Returns the point at the given fraction between a and b, where
-    'fraction' must be between 0 and 1.
-    """
-    return a + (b - a) * fraction
-
-
 def rankdata(a):
     """
     Ranks the data, dealing with ties appropriately.
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -38,7 +38,7 @@
 import pandas.computation.expressions as expressions
 from pandas.computation.eval import eval as _eval
 from pandas.computation.scope import _ensure_scope
-from pandas.compat.scipy import scoreatpercentile as _quantile
+from numpy import percentile as _quantile
 from pandas.compat import(range, zip, lrange, lmap, lzip, StringIO, u,
                           OrderedDict, raise_with_traceback)
 from pandas import compat
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -52,7 +52,7 @@
 import pandas.tslib as tslib
 import pandas.index as _index
 
-from pandas.compat.scipy import scoreatpercentile as _quantile
+from numpy import percentile as _quantile
 from pandas.core.config import get_option
 
 __all__ = ['Series']
@@ -1235,10 +1235,11 @@ def quantile(self, q=0.5):
         valid_values = self.dropna().values
         if len(valid_values) == 0:
             return pa.NA
-        result = _quantile(valid_values, q * 100)
-        if not np.isscalar and com.is_timedelta64_dtype(result):
-            from pandas.tseries.timedeltas import to_timedelta
-            return to_timedelta(result)
+        if com.is_datetime64_dtype(self):
+            values = _values_from_object(self).view('i8')
+            result = lib.Timestamp(_quantile(values, q * 100))
+        else:
+            result = _quantile(valid_values, q * 100)
 
         return result
 


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -10915,13 +10915,13 @@ def wrapper(x):
                             check_dtype=False, check_dates=True)
 
     def test_quantile(self):
-        from pandas.compat.scipy import scoreatpercentile
+        from numpy import percentile
 
         q = self.tsframe.quantile(0.1, axis=0)
-        self.assertEqual(q['A'], scoreatpercentile(self.tsframe['A'], 10))
+        self.assertEqual(q['A'], percentile(self.tsframe['A'], 10))
         q = self.tsframe.quantile(0.9, axis=1)
         q = self.intframe.quantile(0.1)
-        self.assertEqual(q['A'], scoreatpercentile(self.intframe['A'], 10))
+        self.assertEqual(q['A'], percentile(self.intframe['A'], 10))
 
         # test degenerate case
         q = DataFrame({'x': [], 'y': []}).quantile(0.1, axis=0)
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -1907,17 +1907,17 @@ def test_groupby_with_hier_columns(self):
         self.assert_(result.columns.equals(df.columns[:-1]))
 
     def test_pass_args_kwargs(self):
-        from pandas.compat.scipy import scoreatpercentile
+        from numpy import percentile
 
-        def f(x, q=None):
-            return scoreatpercentile(x, q)
-        g = lambda x: scoreatpercentile(x, 80)
+        def f(x, q=None, axis=0):
+            return percentile(x, q, axis=axis)
+        g = lambda x: percentile(x, 80, axis=0)
 
         # Series
         ts_grouped = self.ts.groupby(lambda x: x.month)
-        agg_result = ts_grouped.agg(scoreatpercentile, 80)
-        apply_result = ts_grouped.apply(scoreatpercentile, 80)
-        trans_result = ts_grouped.transform(scoreatpercentile, 80)
+        agg_result = ts_grouped.agg(percentile, 80, axis=0)
+        apply_result = ts_grouped.apply(percentile, 80, axis=0)
+        trans_result = ts_grouped.transform(percentile, 80, axis=0)
 
         agg_expected = ts_grouped.quantile(.8)
         trans_expected = ts_grouped.transform(g)
@@ -1935,7 +1935,7 @@ def f(x, q=None):
 
         # DataFrame
         df_grouped = self.tsframe.groupby(lambda x: x.month)
-        agg_result = df_grouped.agg(scoreatpercentile, 80)
+        agg_result = df_grouped.agg(percentile, 80, axis=0)
         apply_result = df_grouped.apply(DataFrame.quantile, .8)
         expected = df_grouped.quantile(.8)
         assert_frame_equal(apply_result, expected)
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -2137,17 +2137,28 @@ def test_prod_numpy16_bug(self):
         self.assertNotIsInstance(result, Series)
 
     def test_quantile(self):
-        from pandas.compat.scipy import scoreatpercentile
+        from numpy import percentile
 
         q = self.ts.quantile(0.1)
-        self.assertEqual(q, scoreatpercentile(self.ts.valid(), 10))
+        self.assertEqual(q, percentile(self.ts.valid(), 10))
 
         q = self.ts.quantile(0.9)
-        self.assertEqual(q, scoreatpercentile(self.ts.valid(), 90))
+        self.assertEqual(q, percentile(self.ts.valid(), 90))
 
         # object dtype
         q = Series(self.ts,dtype=object).quantile(0.9)
-        self.assertEqual(q, scoreatpercentile(self.ts.valid(), 90))
+        self.assertEqual(q, percentile(self.ts.valid(), 90))
+
+        # datetime64[ns] dtype
+        dts = self.ts.index.to_series()
+        q = dts.quantile(.2)
+        self.assertEqual(q, Timestamp('2000-01-10 19:12:00'))
+
+        if not _np_version_under1p7:
+            # timedelta64[ns] dtype
+            tds = dts.diff()
+            q = tds.quantile(.25)
+            self.assertEqual(q, pd.to_timedelta('24:00:00'))
 
     def test_describe(self):
         _ = self.series.describe()
diff --git a/pandas/tseries/tests/test_timedeltas.py b/pandas/tseries/tests/test_timedeltas.py
--- a/pandas/tseries/tests/test_timedeltas.py
+++ b/pandas/tseries/tests/test_timedeltas.py
@@ -240,7 +240,7 @@ def test_timedelta_ops(self):
 
         result = td.quantile(.1)
         # This properly returned a scalar.
-        expected = to_timedelta('00:00:02.6')
+        expected = np.timedelta64(2599999999,'ns')
         tm.assert_almost_equal(result, expected)
 
         result = td.median()[0]