BUG: COMPAT:0.18 added scipy version check GH12887
State: closed

- [x] closes #12887 
- [X] tests passed (Tests specific to this issue now pass with scipy dev and numpy dev versions that I was using when I opened the issue)
- [X] passes `git diff upstream/master | flake8 --diff`
- [X] whatsnew entry
  @jreback @TomAugspurger 


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Could you also add a test in `pandas/tests/test_generic` for this? Apparently we don't have any that hit `piecewise_polynomial` directly. The behavior should be
- pandas 0.18.1, scipy <= 0.17: `piecewise_polynomial` continues to work, `from_derivatives` now works too
- pandas 0.18.1, scipy > 0.17: `piecewise_polynomial` fails (scipy's error)  `from_derivatives` works

we need a test to assert all of this stuff. put in `test_generic` where the rest of the interpolation stuff already is.

I'll incorporate all of this. Thanks for the feedback thus far!

I'll write the tests once I get the 'okay' on  `missing.py` and `generic.py` code is acceptable. Looking forward to both of your comments on these latest commits. I'll squash everything down once I get it all wrapped up.

always write tests first! that way can validate you are testing the right thing

You're right, should have started there. I'll get to work on those.

@jmwoloso can you update

Update to the most recent pandas dev you mean?

yep

Sure thing. Away from my comp currently but will do when I get back.

Here's a patch which makes this clean. Just needs tests.

```
diff --git a/doc/source/whatsnew/v0.18.1.txt b/doc/source/whatsnew/v0.18.1.txt
index 4b1326a..0ca6683 100644
--- a/doc/source/whatsnew/v0.18.1.txt
+++ b/doc/source/whatsnew/v0.18.1.txt
@@ -512,6 +512,7 @@ Other API changes
 - Provide a proper ``__name__`` and ``__qualname__`` attributes for generic functions (:issue:`12021`)
 - ``pd.concat(ignore_index=True)`` now uses ``RangeIndex`` as default (:issue:`12695`)
 - ``pd.merge()`` and ``DataFrame.join()`` will show a ``UserWarning`` when merging/joining a single- with a multi-leveled dataframe (:issue:`9455`, :issue:`12219`)
+- Compat with SciPy > 0.17 for deprecated ``piecewise_polynomial`` interpolation method (:issue:`12887`)

 .. _whatsnew_0181.deprecations:

@@ -656,5 +657,3 @@ Bug Fixes
 - Bug in ``to_numeric`` with ``Index`` returns ``np.ndarray``, rather than ``Index`` (:issue:`12777`)
 - Bug in ``to_numeric`` with datetime-like may raise ``TypeError`` (:issue:`12777`)
 - Bug in ``to_numeric`` with scalar raises ``ValueError`` (:issue:`12777`)
-- ``pd.read_excel()`` now accepts column names associated with keyword argument ``names``(:issue:`12870`)
-- Compat with SciPy > 0.17 for deprecated 'piecewise_polynomial' interpolation method (:issue:`12887`)
diff --git a/pandas/core/missing.py b/pandas/core/missing.py
index c67fc78..0bf112d 100644
--- a/pandas/core/missing.py
+++ b/pandas/core/missing.py
@@ -83,8 +83,6 @@ def clean_fill_method(method, allow_nearest=False):


 def clean_interp_method(method, **kwargs):
-    import scipy
-    scipy_version = scipy.__version__
     order = kwargs.get('order')
     valid = ['linear', 'time', 'index', 'values', 'nearest', 'zero', 'slinear',
              'quadratic', 'cubic', 'barycentric', 'polynomial', 'krogh',
@@ -96,11 +94,7 @@ def clean_interp_method(method, **kwargs):
     if method not in valid:
         raise ValueError("method must be one of {0}."
                          "Got '{1}' instead.".format(valid, method))
-    # compat GH12887
-    if method == 'piecewise_polynomial' and LooseVersion(scipy_version) > \
-            LooseVersion('0.17'):
-        raise ValueError("Method '{0}' is deprecated for Scipy > 0.17. "
-                         "Use 'from_derivatives' instead".format(method))
+
     return method


@@ -200,7 +194,8 @@ def interpolate_1d(xvalues, yvalues, method='linear', limit=None,

     sp_methods = ['nearest', 'zero', 'slinear', 'quadratic', 'cubic',
                   'barycentric', 'krogh', 'spline', 'polynomial',
-                  'piecewise_polynomial', 'pchip', 'akima']
+                  'from_derivatives', 'piecewise_polynomial', 'pchip', 'akima']
+
     if method in sp_methods:
         inds = np.asarray(xvalues)
         # hack for DatetimeIndex, #1646
@@ -233,7 +228,6 @@ def _interpolate_scipy_wrapper(x, y, new_x, method, fill_value=None,
         raise ImportError('{0} interpolation requires Scipy'.format(method))

     new_x = np.asarray(new_x)
-    scipy_version = scipy.__version__  # for version check; GH12887

     # ignores some kwargs that could be passed along.
     alt_methods = {
@@ -259,13 +253,13 @@ def _interpolate_scipy_wrapper(x, y, new_x, method, fill_value=None,
         except ImportError:
             raise ImportError("Your version of Scipy does not support "
                               "Akima interpolation.")
-    elif method == 'piecewise_polynomial':  # GH12887
-        if LooseVersion(scipy_version) > LooseVersion('0.17'):
-            raise ValueError("Method '{0}' is deprecated for Scipy > 0.17. "
-                             "Use 'from_derivatives' instead".format(method))
+    elif method == 'piecewise_polynomial':
+        """ return the method for compat with scipy version """
+        if LooseVersion(scipy.__version__) <= LooseVersion('0.17'):
+            f = 'piecewise_polynomial_interpolate'
         else:
-            alt_methods['piecewise_polynomial'] = \
-                interpolate.piecewise_polynomial_interpolate
+            f = 'from_derivatives'
+        alt_methods['piecewise_polynomial'] = getattr(interpolate, f)

     interp1d_methods = ['nearest', 'zero', 'slinear', 'quadratic', 'cubic',
                         'polynomial']
```

https://github.com/jreback/pandas/commit/31e01ae5acc4c6ac822d1579ce507f3422663268

xref #13055 moved interpolation tests to a better location.

ok here's a working / tested branch: https://github.com/jreback/pandas/tree/jmwoloso-compat_12887

though the tests work, they don't really exercise this too much (as they same results as using akima). this this is a very simple interp problem though.

Quick question. Should I be using the latest dev branch of scipy and numpy while I'm doing this or perhaps the latest stable releases of each? I'm currently using

```
>>> scipy.__version__
'0.18.0.dev0+609facc'
>>> numpy.__version__
'1.12.0.dev0+47b6c2b'
```

I ask because after applying your patch, I get an `AttributeError` when calling `interpolate` with `method='piecewise_polynomial'`. To fix that, I change line 262 in `pandas.core.missing` from:

```
alt_methods['piecewise_polynomial'] = getattr(interpolate, f)
```

to

```
alt_methods['piecewise_polynomial'] = getattr(interpolate.BPoly, f)
```

This change takes care of the AttributeError, which, I'm guessing was occurring because `BPoly.from_derivatives` is a class method(?)

However, it then raises an error mentioning the expected shape of y:

```
>>> import pandas as pd; import numpy as np
>>> s = pd.Series([0, np.nan, 2])
>>> s.interpolate(method='piecewise_polynomial')
Traceback (most recent call last):
  File "/home/jason/.virtualenvs/pandas_dev/lib/python3.4/site-packages/scipy/interpolate/interpolate.py", line 1452, in from_derivatives
    k = max(len(yi[i]) + len(yi[i+1]) for i in range(m))
  File "/home/jason/.virtualenvs/pandas_dev/lib/python3.4/site-packages/scipy/interpolate/interpolate.py", line 1452, in <genexpr>
    k = max(len(yi[i]) + len(yi[i+1]) for i in range(m))
TypeError: object of type 'numpy.float64' has no len()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/generic.py", line 3576, in interpolate
    **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 2926, in interpolate
    return self.apply('interpolate', **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 2890, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 906, in interpolate
    downcast=downcast, mgr=mgr, **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 969, in _interpolate
    interp_values = np.apply_along_axis(func, axis, data)
  File "/home/jason/.virtualenvs/pandas_dev/lib/python3.4/site-packages/numpy/lib/shape_base.py", line 91, in apply_along_axis
    res = func1d(arr[tuple(i.tolist())], *args, **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 966, in func
    bounds_error=False, **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/missing.py", line 210, in interpolate_1d
    order=order, **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/missing.py", line 288, in _interpolate_scipy_wrapper
    new_y = method(x, y, new_x, **kwargs)
  File "/home/jason/.virtualenvs/pandas_dev/lib/python3.4/site-packages/scipy/interpolate/interpolate.py", line 1454, in from_derivatives
    raise ValueError("Using a 1D array for y? Please .reshape(-1, 1).")
ValueError: Using a 1D array for y? Please .reshape(-1, 1).
```

make sure you have the latest pull of my branch

it works on all versions of scipy and numpy (that code is a previous version)

Ok, will do. I was going crazy for minute...

Pulled down your branch and merged it with mine, though I'm still getting the same errors when using `method='piecewise_polynomial'` BUT `method='from_derivatives'` works as expected.

```
>>> import pandas as pd;import numpy as np
>>> s = pd.Series([0, np.nan, 2])
>>> s.interpolate(method='from_derivatives')
0    0.0
1    1.0
2    2.0
dtype: float64
>>> s.interpolate(method='piecewise_polynomial')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/generic.py", line 3576, in interpolate
    **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 2926, in interpolate
    return self.apply('interpolate', **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 2890, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 906, in interpolate
    downcast=downcast, mgr=mgr, **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 969, in _interpolate
    interp_values = np.apply_along_axis(func, axis, data)
  File "/home/jason/.virtualenvs/pandas_dev/lib/python3.4/site-packages/numpy/lib/shape_base.py", line 91, in apply_along_axis
    res = func1d(arr[tuple(i.tolist())], *args, **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/internals.py", line 966, in func
    bounds_error=False, **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/missing.py", line 210, in interpolate_1d
    order=order, **kwargs)
  File "/home/jason/source/dev/pandas-jmwoloso/pandas/core/missing.py", line 263, in _interpolate_scipy_wrapper
    alt_methods['piecewise_polynomial'] = getattr(interpolate, f)
AttributeError: 'module' object has no attribute 'from_derivatives'
```

I must be doing something wrong, but git says I'm up to date:

```
git pull https://github.com/jreback/pandas.git jmwoloso-compat_12887
From https://github.com/jreback/pandas
* branch            jmwoloso-compat_12887 -> FETCH_HEAD
Already up-to-date.
```

push up your changes and I'll look

Just pushed them. I did have a merge conflict that I had to fix, I was using `getattr(interpolate.BPoly, f)` in the `alt_methods` assignment in `_interpolate_scipy_wrapper`  before in order to get it to use `from_derivatives`. Which then caused the shape error, but I see you've fixed that with the `_from_derivatives` convenience function which is good. I was going to tackle that, but I never use these interpolation methods, so it definitely would have taken me longer to figure out what it was expecting exactly.

yeah it looks like you have some older code here.

I am going to merge the branch I had earlier. Would appreciate if you can push some better tests though.

Yeah, I'll try rebasing with master again as well, I don't have `pandas.tests.test_missing` at all and I should

@jmwoloso sorry for the rush on this. trying to get the release out the door tomorrow :).

If you can push some tests that would be great (no big hurry on those though).

Understand completely. I appreciate the help. I'll get those tests pushed once I see what you've written already for them.

great! I just copied the akima exactly. It gives the same results, so might it might be right. But I'ld ideally like something better. (but don't go crazy).

Haha, sounds good. I was eyeballing akima as well. I'll see what else I can come up with. Thanks again! This was fun (my git-fu needs lots of work still). I'll browse for some other issues to tackle once I get these tests done.