Implement DataFrame.__array_ufunc__
State: closed

For some cases, this will preserve extension types of arrays by calling
the ufunc blockwise.

```python
In [1]: import pandas as pd; import numpy as np
In [2]: df = pd.DataFrame({"A": pd.array([0, 1], dtype="Sparse")})

In [3]: np.sin(df).dtypes
Out[3]:
A    Sparse[float64, nan]
dtype: object
```

Implementation-wise, this was done by moving `Series.__array_ufunc__` to `NDFrame` and making it generic for Series / DataFrame. The DataFrame implementation goes through `BlockManager.apply(ufunc)`.

We don't currently handle the multi-input case well for dataframes (aside from ufuncs that are implemented as dunder ops like `np.add`). For these, we fall back to the old implementation of converting to an ndarray and wrapping the result. This loses extension types.

We also don't currently handle multi-output ufuncs (like `np.modf`). This would require a `BlockManager.apply` that returns a `Tuple[BlockManager]`, `nout` per input block. Maybe someday, but that's low priority.

closes https://github.com/pandas-dev/pandas/issues/23743

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
needs rebase, otherwise i think pretty ready
can you merge master
@TomAugspurger needs rebase, otherwise i think good to go
if we were to implement `__array_function__` (after dropping numpy 1.16 i guess) is that a "instead of" or an "in addition to" `__array_ufunc__` thing?

does this make `__array_wrap__` unnecessary?
`__array_ufunc__` is separate from `__array_function__`. If there's ever a specialized protocol (like `__array_ufunc__`) it takes priority over `__array_function__`.

IIRC I tried removing `__array_wrap__` but there may have been a test failure. Apparently it's used by things other than `__array_ufunc__`.
@TomAugspurger can you merge master?  i tried doing it myself but am getting some really weird-looking errors
Gonna be a bit probably.

> On Nov 20, 2020, at 2:02 PM, jbrockmendel <notifications@github.com> wrote:
> 
> 
> @TomAugspurger <https://github.com/TomAugspurger> can you merge master? i tried doing it myself but am getting some really weird-looking errors
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/pandas-dev/pandas/pull/36955#issuecomment-731380237>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOISEIYDAE3HVNRKCPODSQ3DMRANCNFSM4SHYJLYQ>.
> 


ideally we could do this for 1.2 if can merge master
@jreback i merged master 2 days ago, this should be OK
thanks @TomAugspurger very nice 
Thanks Brock. 
The fact that we now align is a breaking change, but it seems one that could easily be detected (when the operands are not yet aligned) and done with a deprecation cycle?
Did we face a similar issue with Series implementing `__array_ufunc__` in https://github.com/pandas-dev/pandas/pull/23293? I don't see anything in the change log?

I vaguely recall a discussion around `np.add(a, b)` aligning, but, say, `np.logaddexp(a, b)` not, being considered a bug.
Yes, we did a similar change back then for Series I think, and it seems we didn't discuss it much at the time either.  And I also don't recall (m)any issues about it that were raised after doing the change. Which is certainly a sign that this might be indeed OK to simply change.

I would say the main differences is that for Series we changed this in 0.25, but now for 1.x, we said to do changes using a deprecation cycle where possible (and this seems a case where it is easily possible)


Right, the 1.0 vs. 0.25 distinction is the important part. I'd prefer to see a deprecation warning for the alignment, but I don't know if I'll have time to do it for this release.

One complication just noticed: there isn't an easy way to silence the warning, like with a keyword. We'd need to only warn when the inputs are unaligned, and require users to call `.align()` ahead of time.
> One complication just noticed: there isn't an easy way to silence the warning, like with a keyword. We'd need to only warn when the inputs are unaligned, and require users to call .align() ahead of time.

Indeed, aligning manually would be the way to get around the warning if you want the new behaviour, otherwise the solution is to first convert to numpy arrays before calling the ufunc, in case you want the old non-aligning behaviour.