COMPAT: Expand compatibility with fromnumeric.py
State: closed

Fixes up OR addresses what are AFAICT almost if not all of the incompatibilities `pandas` currently has with `numpy.core.fromnumeric`.  My goodness, were there a lot of them.

Closes #12638.
Closes #12644.
Closes #12687.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
ok, some comments.

I hate to be a curmudgeon on this issue, but how do we benefit from increasing our API coupling to NumPy? We've been discussing moving precisely in the opposite direction. 

@wesm in recent versions of numpy (like > 1.11), they are now wrapping all common functions that try to dispatch to 'duck-like' class that can handle (w/o directly using) `__array__`, so sub-classing (and duck-like behavior is preservered). not a bad thing actually. The problem is pandas has a different signature and this is not handled nicely (by numpy).

```
In [1]: p = pd.SparseDataFrame([1, 2, 3])

In [2]: np.cumsum(p)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-74227eeed55a> in <module>()
----> 1 np.cumsum(p)

/Users/jreback/miniconda/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in cumsum(a, axis, dtype, out)
   2113     except AttributeError:
   2114         return _wrapit(a, 'cumsum', axis, dtype, out)
-> 2115     return cumsum(axis, dtype, out)
   2116 
   2117 

TypeError: cumsum() takes at most 2 arguments (4 given)
```
- so we are forced to either accept some incompatibilities
- outlaw this
- ignore it
- accomodate it by 'accepting' these arguments and ignoring them (while not showing them in the pandas signature). This is exacerbated a bit by numpy passing positional args, rather than kwargs.

it IS unfortunate, and maybe numpy should just catch the `TypeError` around the `_wrap_it`.

I think it is nicer to make it just work.

@jreback : indeed, as I believe you saw with my first PR for `fromnumeric.py` against `numpy`, `TypeError` exceptions will now be caught, though it needs a little tweaking to make sure the only ones caught are signature incompatibilities (see my most recent PR).

@gfyoung what's the PR number?

<a href="https://github.com/numpy/numpy/pull/7425">#7425</a> (first PR, merged)
<a href="https://github.com/numpy/numpy/pull/7491">#7491</a> (second PR, open)

FYI: all the failures that I have currently against `numpy-dev` on Travis are fixed with the second PR.

so when that 2nd one is merged, why do we need this PR? (or the .round fixes)?

First, for backwards compatibility, this PR is important in helping remove signature incompatibilities between the `pandas` library and `numpy`.  In the long run, this PR will help to ensure that the `pandas` implementation gets called over the `numpy` one.  For example, `Categorical.transpose` returns `self`, so `np.transpose(Categorical)` should do the same.

but you are changing LOTS of functions

Am I though?  Adding `*args` and `**kwargs` to signatures should not affect anyone's current code, and the validation should not be problematic if people are respecting the signatures and not passing in nonsense arguments.

Also, I will point out that this PR helps in terms of removing extraneous `numpy`-compatible arguments from signatures and hiding them away in `*args` and `**kwargs` parameters.

@gfyoung I am not averse to having compat, just since numpy broke it, it makes sense for them to fix it.

@jreback : Agreed.  That's why I'm fixing the issue on that end too.  As I might have mentioned before, `scipy` also has compat issues with this same module.  So it's not just `pandas`.

ok, so let's table this till then. I guess the `.round()` fix needs to stay though as numpy 1.11 broke it (because of the positional issues?) or maybe 1.11.1 will fix this?

@jreback : I would contend that this PR is useful for people who are using older versions of `numpy`.  In addition, the `round` fix on the `pandas` end is included in this PR.  The fix on the `numpy` end is not sufficient on its own, as I commented in that PR.  It's the combination of this PR and that one that will fix the failing test.

Having worked on both ends of this issue, I think this should still be tagged for `0.18.1`.

can you run just the tests (or this PR) (no fix) on 1.11 and 1.10 and show results (the errors)

@jreback : I don't quite know how I can run _all_ of the tests from this PR on the `master` branch, but I think just running the unit tests in `pandas/tests` and `pandas/sparse/tests` speaks for itself:

[numpy-v1.11.0-sparse-tests.txt](https://github.com/pydata/pandas/files/208802/numpy-v1.11.0-sparse-tests.txt)
[numpy-v1.11.0-tseries-tests.txt](https://github.com/pydata/pandas/files/208943/numpy-v1.11.0-tseries-tests.txt)
[numpy-v1.11.0-tests.txt](https://github.com/pydata/pandas/files/208884/numpy-v1.11.0-tests.txt)
[numpy-v1.10.4-sparse-tests.txt](https://github.com/pydata/pandas/files/208893/numpy-v1.10.4-sparse-tests.txt)
[numpy-v1.10.4-tseries-tests.txt](https://github.com/pydata/pandas/files/208944/numpy-v1.10.4-tseries-tests.txt)
[numpy-v1.10.4-tests.txt](https://github.com/pydata/pandas/files/208913/numpy-v1.10.4-tests.txt)

@gfyoung I just want you to run THIS PR tests, the ones you added.

@jreback : ah, fair enough, but the error log shows you that most of my tests fail.

can you post a summary (e.g. show a list of the failing functions)

`repeat`
`take`
`transpose`
`argsort`
`sort`
`round`
`clip`
`cumsum`
`argmin`
`argmax`
`min`
`max`

Essentially any function that needs defaults (either in the existing `pandas/compat/func_defaults.py` or in the future `pandas/compat/numpy/function.py`) is **with high probability** a function that will fail _right now_ (for multiple classes too I might add), less those whose extraneous `numpy`-compat arguments I removed.

@jreback : I've done all of the requested refactoring, so now all of the validation is hidden from the main modules and now resides in `pandas/compat/numpy/function.py`.  Travis is passing, and the remaining two failures on `numpy-dev` should be fixed with my `numpy` PR <a href="https://github.com/numpy/numpy/pull/7491">#7491</a>.

@gfyoung make this smaller, e.g. pull out extraneous things. There is too much going on and too many moving parts. 

If this is going to break user code in 0.18.x, then we should make trunk 0.19.x and create a 0.18.x maintenance branch -- can't keep breaking code in minor releases.

I'm for NumPy compatibility as a matter of convenience, but not a contract with the user. So we should not be saying "we are guaranteeing that pandas.Series and DataFrame are interchangeable with ndarray".

In general, I don't think we should be encouraging more ndarray duck typing (for example, indexing will always behave a little differently because of indexes) -- we need to be able to control the behavior and semantics of pandas without being as dependent on NumPy as we have in the past. I even think we should provide our own ufunc wrappers and deprecate usage of `np.{exp, sqrt, log, ...}`. 

In particular, we definitely should not be making any changes that require an upgrade to bleeding edge NumPy. e.g. just digging around a little bit https://github.com/numpy/numpy/commit/fdb05497a1501dc84c7942d85155b4cf4b22df95

I don't think that pandas should be under any obligation to conform its API to NumPy's. I find nowadays that most pandas users make very little direct use of NumPy anymore. In 2008, the world was very different, so making pandas more NumPy-like helped us. Making extra efforts now seems anachronistic. 

@wesm: I agree with most of your comments, but this PR is not about **aligning** the `pandas` API wiht `numpy`'s (in fact, nowhere in this PR or in the conversation, did we mention anything about being "forced" to align the API with `numpy`'s)  It is about **accommodating** that API, as there are still users, as evidenced by the issues I am addressing, who attempt to use `numpy` functions on `pandas` objects.  I'm not adding new arguments besides `*args` and `**kwargs` to signatures and actively discouraging passage of `numpy`-extraneous parameters with the argument validation.

@jreback : Of course there would be a lot of "moving parts": `fromnumeric.py` is a LARGE module, and there are A LOT of incompatibilities right now.  Breaking up this PR would largely amount to breaking up the resolution of these incompatibilities.

Also, I don't quite agree with the implication (at least as I interpret from your statement of "moving parts") that I am changing that much.  Most of these changes are test additions because there was surprisingly not a lot of test coverage (or rather, I could not find it) for some of the functions I was addressing.  Does adding `*args` and `*kwargs` to a bunch of functions really constitute something that impactful to the code base?

@jreback : I've successfully addressed all of the changes you have requested, with the exception of this <a href="https://github.com/pydata/pandas/pull/12810#discussion_r59142791">one<a/> and this <a href="https://github.com/pydata/pandas/pull/12810#discussion_r59142700">one<a/>.  I was hoping to get further comment on those two.

@jreback : I have successfully addressed all of the changes you have requested, including now the two remaining ones I mentioned above.

what is the status on the numpy change? I'd like to have that merged before we do this