COMPAT: remove compat.scipy
State: closed


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
is this used anywhere? might be older code

This is only an internal function (copied exactly from scipy https://github.com/scipy/scipy/blob/master/scipy/stats/stats.py#L4482), and the only use is in the function above (https://github.com/pydata/pandas/blob/master/pandas/compat/scipy.py#L35), which in turn is only used in some tests I think.  
So I don't if it is worth to adapt this.

@clham if you'd like to make `fastsort` internal to `rankdata` would be ok. these routines are really  just to check that the pandas routines track scipy and are not used anywhere outside the test suite AFAIK.

@jorisvandenbossche think we just ought to move them to pandas.util.testing? (and take out compat.scipy)

@jreback maybe that's a good idea, since they are only for testing to put them there.

@clham if you want to do a PR to:

put these functions into `pandas.util.testing` (and fold `fastsort` into `rankdata`), then change the tests to use this (essentially remove `compat.scipy`)

@jreback the `percentileofscore` is still used in pandas/stats/misc (although I don't know if that last is really used)

@jorisvandenbossche hmm, I see that.

ok well `percentileRank` should be changed to `percentile_rank` (and I don't see it used anywhere)

ok..then just let's take out `rankdata` and `fastsort` (and put in testing)

I would even be in favor of making `percentile_rank` requiring scipy; we use it in a lot of places (and mainting compat is something we shouldn't really be doing)

what do you think?

@jreback I wouldn't change the name of `percentileRank`, it is nowhere used or publicized in pandas, but if there would be someone using it, not worth the trouble to change it I think (unless we want to add it to the public/publicized api, we should rather deprecate/remove it I think).

But requiring scipy does not seem a problem, then compat.scipy can just be removed.

ok @clham 

so idea is to completely remove `compat.scipy`
- move `fastsort` inside `rankdata`
- move `rankdata` to `pandas.util.testing`, maybe rename to `compat_rankdata`
- change tests in `test_frame/test_series/test_tseries` to use `compat_rankdata` rather than `rankdata`
- change `percentileRank` to use `scipy` directly (not sure this is tested at all, btw)

hopefully you can submit a PR for this!

Sure. I'll see what I can do.

@jreback 

I'm confused here.  

1) Why can't we convert  `rankdata` to use `scipy` directly?
2)  `percentilerank` can't use scipy directly, because the current version of `scipy` removed the options for tiebreaking.  

Was this a typo, or am I missing the ball? I'm too new to assume much at this point.

(also, percentileRank behaves erratically when fed a 2d(+) dataframe, but that is an issue for after everything is moved around) 

@clham you can try removing rankdata entirely (and just import from scipy). that might just work. (i think the original intent was to allow the tests to work w/o scipy). but its included on almost all builds now. you might need to use `_skip_if_no_scipy()` in that particular function where its tested (in test_series.py), so it skips that build