compatibility with scipy 0.19
State: closed

fix #15662

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
note that there are more skips

```
pandas/tests//frame/test_missing.py:        tm.skip_if_no_package('scipy', max_version='0.19.0')
pandas/tests//series/test_missing.py:        tm.skip_if_no_package('scipy', max_version='0.19.0')
pandas/tests//test_window.py:        tm.skip_if_no_package('scipy', max_version='0.19.0')
pandas/tests//test_window.py:        tm.skip_if_no_package('scipy', max_version='0.19.0')
pandas/tests//test_window.py:        tm.skip_if_no_package('scipy', max_version='0.19.0')
pandas/tests//test_window.py:        tm.skip_if_no_package('scipy', max_version='0.19.0')
pandas/tests//test_window.py:        tm.skip_if_no_package('scipy', max_version='0.19.0')
```
This ones ok
```
pandas/tests//sparse/test_frame.py:    tm.skip_if_no_package('scipy', max_version='0.19.0')
```

if any remain (IOW can't / not easy fix), then we should switch these to xfailing (in 0.19.0), but that's a bit more tricky.
@jreback 

> note that there are more skips

I've fixed 5 of them, and plan to fix 2 more.

`pandas/tests//sparse/test_frame.py` never appears in my original issue. So not sure what to do with it.
@zym1010 sure np. Yeah ignore the ``sparse`` one; I added that because of an incompat with certain types of sparse matrices in 0.19.0 (so not relevant to this issue; it was another change in scipy 0.19.0).
@jreback function-wise, it's now fixed. I will fix the comment later.
looks like just a couple of linting issues

``git diff master | flake8 --diff`` will show them locally.

add a whatsnew and looks good to go.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15689?src=pr&el=h1) Report
> Merging [#15689](https://codecov.io/gh/pandas-dev/pandas/pull/15689?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/2cad4dd0b48946add99d3d90e3dba958f2885349?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master   #15689      +/-   ##
==========================================
- Coverage   91.02%      91%   -0.03%     
==========================================
  Files         143      143              
  Lines       49409    49409              
==========================================
- Hits        44977    44965      -12     
- Misses       4432     4444      +12
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15689?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/window.py](https://codecov.io/gh/pandas-dev/pandas/compare/2cad4dd0b48946add99d3d90e3dba958f2885349...3cc6528fe70a815d85849fd5a9f534e672c0d701?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvd2luZG93LnB5) | `96.18% <100%> (ø)` | :white_check_mark: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/compare/2cad4dd0b48946add99d3d90e3dba958f2885349...3cc6528fe70a815d85849fd5a9f534e672c0d701?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :x: |
| [pandas/tools/merge.py](https://codecov.io/gh/pandas-dev/pandas/compare/2cad4dd0b48946add99d3d90e3dba958f2885349...3cc6528fe70a815d85849fd5a9f534e672c0d701?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL21lcmdlLnB5) | `93.68% <0%> (-0.34%)` | :x: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/compare/2cad4dd0b48946add99d3d90e3dba958f2885349...3cc6528fe70a815d85849fd5a9f534e672c0d701?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `80.66% <0%> (-0.19%)` | :x: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/compare/2cad4dd0b48946add99d3d90e3dba958f2885349...3cc6528fe70a815d85849fd5a9f534e672c0d701?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.87% <0%> (-0.1%)` | :x: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/compare/2cad4dd0b48946add99d3d90e3dba958f2885349...3cc6528fe70a815d85849fd5a9f534e672c0d701?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `91.3% <0%> (+0.33%)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15689?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15689?src=pr&el=footer). Last update [2cad4dd...3cc6528](https://codecov.io/gh/pandas-dev/pandas/compare/2cad4dd0b48946add99d3d90e3dba958f2885349...3cc6528fe70a815d85849fd5a9f534e672c0d701?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
lgtm. ping on green.
thanks @zym1010 
hmm, seems https://travis-ci.org/pandas-dev/pandas/jobs/211720204 is showing a failure on *master* scipy that does not show up on 0.19.0 :<

to test installing these look at ``ci/requirements-3.5_NUMPY_DEV.build.sh`` we are installing from wheels (so you have to create a separate environment to build this).


hmm, maybe they changed it back? (it looks like its compute the < 0.19.0) one? weird
@jreback no idea. I think for scipy 0.19, definitely it should not return the pre 0.19 result. I think it works with conda-forge version of Scipy. What's the difference between scipy 0.19 dev and that one?
I wonder if @ev-br has any idea on this.
Did they? That's bizzare. I can take a look --- a self-contained example would help though. 
@ev-br 

first one (copied from #15662), the expected result of scipy 0.19, is obtained using conda-forge built Scipy. This was tested on an unpatched pandas.

~~~
FAIL: test_interp_scipy_basic (pandas.tests.series.test_missing.TestSeriesInterpolateData)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/tests/series/test_missing.py", line 711, in test_interp_scipy_basic
    assert_series_equal(result, expected)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1181, in assert_series_equal
    obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 59, in pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)
  File "pandas/src/testing.pyx", line 173, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Series are different

Series values are different (33.33333 %)
[left]:  [1.0, 3.0, 6.82352941176, 12.0, 18.0588235294, 25.0]
[right]: [1.0, 3.0, 6.769231, 12.0, 18.230769, 25.0]
~~~

Then the one @jreback just discovered (copied from last part of <https://travis-ci.org/pandas-dev/pandas/jobs/211720204>), is probably obtained on dev version of scipy. This is tested on a patched pandas.

~~~
>       raise AssertionError(msg)
E       AssertionError: Series are different
E       
E       Series values are different (33.33333 %)
E       [left]:  [1.0, 3.0, 6.76923076923, 12.0, 18.2307692308, 25.0]
E       [right]: [1.0, 3.0, 6.823529, 12.0, 18.058824, 25.0]
~~~

Left result is always the result computed by scipy, and right result is the expected result hardcoded in test suite.
> probably obtained on dev version of scipy

Hmmm, can you cook up an example which just makes scipy calls? Will be easier for me to not have to navigate through the pandas codebase which I'm not familiar with.
> Hmmm, can you cook up an example which just makes scipy calls? Will be easier for me to not have to navigate through the pandas codebase which I'm not familiar with.

@jreback can you help with this? As I'm not sure how you set up your scipy environments.
this is from defaults (numpy 1.12) channel. We are also installing 0.19.0 on conda-forge (both these match as below).

The fail is from 
```
numpy: 1.13.0.dev0+dc04055
scipy: 0.19.0.dev0+ce24da1

# install numpy wheel from master
pip install --pre --upgrade --no-index --timeout=60 --trusted-host travis-dev-wheels.scipy.org -f http://travis-dev-wheels.scipy.org/ numpy scipy
```

```
In [9]: import scipy

In [10]: scipy.__version__
Out[10]: '0.19.0'

In [11]: from scipy import interpolate

In [12]: interpolate.interp1d(np.array([0, 1, 3, 5]), np.array([  1.,   3.,  12.,  25.]), kind='quadratic', fill_value=np.nan, bounds_error=False)(np.array([2, 4]))
Out[12]: array([  6.82352941,  18.05882353])

```
Seems same for a -dev version:

```
In [4]: interpolate.interp1d(np.array([0, 1, 3, 5]), np.array([  1.,   3.,  12.,  25.]), kind='quadratic', fill_value=np.nan, bounds_error=False)(np.array([2, 4]))
Out[4]: array([  6.82352941,  18.05882353])

In [5]: import scipy

In [6]: scipy.__version__
Out[6]: '1.0.0.dev0+f3e4e17'
```
http://travis-dev-wheels.scipy.org/

hmm these are really old for scipy! are they being uploaded? 
Ah. Might be that scipy CI only uploads dev wheels in after_success (https://github.com/scipy/scipy/blob/master/.travis.yml#L142) and the builds with numpy-dev fail for the last couple of days because of a numpy-dev regression.
just going to remove it from that build: https://github.com/pandas-dev/pandas/pull/15699
@ev-br hmm, there should be older ones though right? (well not *that* old).
Indeed. https://github.com/scipy/scipy/issues/7188