CI: Troubleshoot CI
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Tacked some CI-troubleshooting stuff onto the just-started FIXMES21 branch.

Labels:
name: CI
description: Continuous Integration

Comments:
great lmk when ready
Looks like npdev changed ndarray.mean s.t. `np.arange(10).view('m8[ns]').mean()` returns a float instead of a td64 obj cc @seberg 
Wow, yet another weird case :(.  Our `mean` code dares to force cast to `"f8"` for timedeltas, so we effectively call:
```
arr.sum(dtype="f8")
```
but, happy coincidence apparently the old datetime resolution is broken and just ignores the dtype request...  Although, I am still surprised why it doesn't just fail in the new code.

Well, I guess, `mean` is arguably broken, if I get luck reverting https://github.com/numpy/numpy/issues/20544#issuecomment-989048601 will "fix" this too (i.e. revert to the old borked up state).
@jbrockmendel if you can fix this up
waiting for a fix in npdev
Should be fixed, not sure if the nightly is build though.
> waiting for a fix in npdev

ok if not fixed, then can you disable tests?
will re-push and find out
Looks like the td64.mean may have gotten fixed but not we're seeing a bunch of new warnings (guessing division by zero related).  will dig into this in a bit
:(, sorry about this, its been a confusing can of worms.  One thing that could still be changed that if you provide `out=` to a reduction, you may end up with a more precise loop (which I suppose could mean float if the array is an integer, but the out is a float).

But I guess this is something awfully gone wrong, or just a different error?:
```
numpy.core._exceptions._UFuncBinaryResolutionError: ufunc 'divide' cannot use operands with types dtype('int8') and dtype('<m8[ns]')
```
Where apparently the type resolution is somehow failing (which confuses me a lot, because I am not sure why this would have been any better with the previous version.)
> But I guess this is something awfully gone wrong, or just a different error?:

Yah in these cases we're currently testing that we get a TypeError, so getting a numpy.core._exceptions._UFuncBinaryResolutionError instead isn't the worst thing in the world; if it can't/won't be changed upstream then we can try to catch and re-raise
Ah OK, puh, I was seriously worried it was not an error before...  That error should be inheriting from `TypeError`, so I think I would prefer to just ignore it, at least from the NumPy side.  But please complain if you think that is annoying.
I think the exception is fine.  The failing tests look like a new warning which im assuming is division by zero, which is also pretty reasonable so we can adjust to handle on our end.
Cool, thanks. Let me know if something is looking shady there, so I can have a closer look.
@jbrockmendel if you have a chance to fix up here (i don't care if you xfail these tests) but cannot merge anything else until we are back to green.
its gonna take a couple more rounds on the CI bc i cant get npdev working locally
It looks very tentatively like a lot of `np_obj.__op__(pd_obj)` calls are going through `pd_obj__array_ufunc__` instead of returning NotImplemented, which we want so they can use `pd_obj.__rop__(np_obj)`
maybe worthile to simply fix the numpy dev wheel to a working one while this is fixed?
> maybe worthile to simply fix the numpy dev wheel to a working one while this is fixed?

first should confirm with @seberg whether the guess about `__array_ufunc__` is correct, and if so if it is something they intend to fix
sorry what i mean is pin the wheel to a working one from 2 days ago
i guess.  no idea how to do that
``numpy==1.23.0.dev0+101.ga81535a36`` in the yaml files?
@lithomas1 can you pinch-hit on this one?
Sure, #44834 is my attempt. We should still keep this PR open to revert the pin in there and come up with a long-term fix.
sounds good.  ive confirmed locally that the td64 object is not respecting `__array_priority__` on numpy:main
@jbrockmendel sorry, only catching up now/tomorrow.  I cannot think of immediate changes related to `__array_priority__` or binary op's behaviour.  This is for numpy's timedelta64 scalars?
(I guess one thing I am confused, is because if `__array_ufunc__` is defined, I believe the design is that the binary op's go through `__array_ufunc__` and never return NotImplemented, unless `__array_ufunc__` is `None`?)
Ah, I had looked at the wrong CI run yesterday, I think.  But I am still confused what the issue is?  Do you have an example?

Btw. assuming you are on linux, one quick hack that may work to get numpy dev, is:
```
python3 runtests.py --shell
```
to drop in a shell with numpy dev version injected into the PYTHONPATH.
I got npdev working locally by installing from source instead of using the wheel.  So I can confirm that a) the `timedelta64.__truediv__` method is not returning NotImplemented as I think it should and b) that actually isn't new (https://github.com/numpy/numpy/issues/17017), so isn't the explanation here.

Found a likely candidate for a bug in our `__array_ufunc__` implementation.  just pushed.