DEPS: Test NEP 50
State: closed

closes #56563

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Blocker for rc
description: Blocking issue or pull request for release candidate

Comments:
@seberg could use your insight to with this behavior

```python
In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: np._set_promotion_state("weak")

In [11]: lib.maybe_convert_objects(np.array([np.int64(0)], dtype=object))
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
Cell In[11], line 1
----> 1 lib.maybe_convert_objects(np.array([np.int64(0)], dtype=object))

File ~/pandas/_libs/lib.pyx:2601, in pandas._libs.lib.maybe_convert_objects()
   2599 
   2600                 if ((seen.uint_ and seen.sint_) or
-> 2601                         val > oUINT64_MAX or val < oINT64_MIN):
   2602                     seen.object_ = True
   2603                     break

OverflowError: Python int too large to convert to C long
```

IIRC `oUINT64_MAX` and `oINT64_MIN` are originally C Ints

https://github.com/pandas-dev/pandas/blob/984d75543fce5c68f0dcc4f0b3500256e4a9c0c9/pandas/_libs/util.pxd#L10-L14

being cast to Numpy ints. 


https://github.com/pandas-dev/pandas/blob/984d75543fce5c68f0dcc4f0b3500256e4a9c0c9/pandas/_libs/lib.pyx#L143-L145

I am not exactly sure, but is `val` a numpy int64 being cast here?
@mroeschke the problem is promotion, normally promotion will convert the python int to the existing version in binary ops.  That said, the NumPy dev branch has additional logic in place to allow comparisons to always succeed.
> That said, the NumPy dev branch has additional logic in place to allow comparisons to always succeed.

OK great! This is definitely more convenient behavior
Also, if we have a dependency (pytables) that uses non-NEP 50 compatible behavior and we make pandas NEP 50 compatible, would you recommend using `_set/get_promotion_state` around the calls that rely on old promotion rules?
OK, I went and dug into a couple tests and it looks like we might be failing some tests because of the new rules such as uint64 * int64 = float64.

I guess we could fix it by trying to cast to the Series/Index dtype inside arithmetic operations.
> the new rules such as uint64 * int64 = float64

Just to note, that rule is not new (everyone hates it).  What is new, though is that if you have have uint64 or int64 be a 0-D object, it was previously cast.  Which is what you still get, but only if it is a Python integer.
> > the new rules such as uint64 * int64 = float64
> 
> Just to note, that rule is not new (everyone hates it). What is new, though is that if you have have uint64 or int64 be a 0-D object, it was previously cast. Which is what you still get, but only if it is a Python integer.

That makes sense, the failing test I was looking at was doing something like 
```
pd.Index([1,2,3], dtype="uint64") * np.int64(3)
```

This is kind of a funky case. Normally, I think it might be OK to just follow the NEP 50 changes (and change the tests), since I think users will also expect Index/Series arithmetic to follow numpy semantics. As Sebastian mentions above, you already get float64 if you multiply by something like ``np.array([3], dtype="int64")

The only issue here, though is that the Arrow backed Indexes will still do the old behavior of trying to safe cast, so we'd be diverging in terms of behavior there, and I'd really like to keep everything consistent.

Here's a little demonstration of the differences
```
>>> import numpy as np
>>> import pandas as pd
>>> pd.Index([1,2,3], dtype="uint64") * np.int64(3)
Index([3.0, 6.0, 9.0], dtype='float64') # This makes the tests fail
>>> pd.Index([1,2,3], dtype="uint64").astype("uint64[pyarrow]") * np.int64(3)
Index([3, 6, 9], dtype='int64[pyarrow]') # Note how pyarrow preserves the int dtype
>>> pd.Index([1,2,3], dtype="uint64") * np.array([3], dtype="int64")
Index([3.0, 6.0, 9.0], dtype='float64') # This happens even in older numpys
>>> pd.Index([1,2,3], dtype="uint64").astype("uint64[pyarrow]") * np.array([3], dtype="int64")
# This doesn't work in pyarrow
```

@pandas-dev/pandas-core thoughts?
There are a lot of options here all with different merits and I'm not sure there will be a clear solution. 

Are those pyarrow results documented behavior? Even the uint64 * int64 resulting in int64 is surprising to me. I would have expected to follow C type promotion rules and end with a uint64
Just noting down other behavior differences, we may want to summarize this in a whatsnew later.

In numpy 2.0,
```
>>> np.float32(1.1) == 1.1
np.True_
```
in numpy 1,
```
>>> np.float32(1.1) == 1.1
False
```

This causes inserting something like 1.1 into a float32 Index to preserve its dtype.
Getting close now,

CI is still red (down to only a couple), but numpy dev is green at least. I still need to tune some of the conditions, but this should be basically ready for a first pass - I'm a little worried about some of the changes I've hacked in.
Finally green.

@mroeschke you might want to take a look again.

also cc @MarcoGorelli since this might affect some PDEP6 stuff.
OK, added more tests. Looks like the one case (that I've caught), where my hack doesn't work is 0, I special cased that out separately.
Will put this in for the RC. Happy to address review comments in a follow up.

Thanks for the reviews everyone!
followup PR created @ #56682