CI: Test on Cython 3.0 on numpydev
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is what we used to do before the libreduction saga. Given that, we finally killed libreduction, I'm bring this back so we can gauge our Cython 3 readiness.

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Hmmm. Looks like hitting https://github.com/cython/cython/issues/2056. Will investigate later.
Hm. I got further this time but the compiled cython module is segfaulting. This is the output from lldb.
Running
``lldb -- python -m pytest pandas/tests/groupby/test_quantile.py -v -k "test_quantile_out_of_bounds_q_raises"`` (and ``run`` in the lldb shell)
I see
<Details>
``
    frame #5: 0x0000000119ceba48 groupby.cpython-38-darwin.so`__pyx_fatalerror(fmt=<unavailable>) at groupby.c:110053:5 [opt]
    frame #6: 0x0000000119d6b977 groupby.cpython-38-darwin.so`__pyx_fuse_3__pyx_pw_6pandas_5_libs_7groupby_107group_quantile [inlined] __Pyx_XCLEAR_MEMVIEW(memslice=<unavailable>, have_gil=1, lineno=40963) at groupby.c:110120:9 [opt]
    frame #7: 0x0000000119d6b965 groupby.cpython-38-darwin.so`__pyx_fuse_3__pyx_pw_6pandas_5_libs_7groupby_107group_quantile [inlined] __pyx_pf_6pandas_5_libs_7groupby_106group_quantile(__pyx_self=<unavailable>, __pyx_v_out=<unavailable>, __pyx_v_values=0x0000000000000000, __pyx_v_labels=0x0000000000000000, __pyx_v_mask=0x0000000000000000, __pyx_v_sort_indexer=<unavailable>, __pyx_v_qs=__Pyx_memviewslice @ 0x0000600002f57560, __pyx_v_interpolation=0x0000000000000000) at groupby.c:40963 [opt]
    frame #8: 0x0000000119d6b657 groupby.cpython-38-darwin.so`__pyx_fuse_3__pyx_pw_6pandas_5_libs_7groupby_107group_quantile(__pyx_self=<unavailable>, __pyx_args=<unavailable>, __pyx_kwds=<unavailable>) at groupby.c:39838 [opt]
    frame #9: 0x00000001088c80dd interval.cpython-38-darwin.so`__pyx_FusedFunction_call [inlined] __Pyx_CyFunction_Call(func=<unavailable>, arg=<unavailable>, kw=<unavailable>) at interval.c:112155:12 [opt]
    frame #10: 0x00000001088c80ca interval.cpython-38-darwin.so`__pyx_FusedFunction_call [inlined] __pyx_FusedFunction_callfunction(func=0x0000000119b119e0, args=<unavailable>, kw=<unavailable>) at interval.c:112651 [opt]
    frame #11: 0x00000001088c80b0 interval.cpython-38-darwin.so`__pyx_FusedFunction_call(func=0x0000000119b119e0, args=0x0000000136b331c0, kw=0x0000000136b59800) at interval.c:112706 [opt]
``
</Details>
in the backtrace.
typing ``f 7``(for frame 7), i see
<Details>

groupby.cpython-38-darwin.so was compiled with optimization - stepping may behave oddly; variables may not be available.
frame #7: 0x0000000119d6b965 groupby.cpython-38-darwin.so`__pyx_fuse_3__pyx_pw_6pandas_5_libs_7groupby_107group_quantile [inlined] __pyx_pf_6pandas_5_libs_7groupby_106group_quantile(__pyx_self=<unavailable>, __pyx_v_out=<unavailable>, __pyx_v_values=0x0000000000000000, __pyx_v_labels=0x0000000000000000, __pyx_v_mask=0x0000000000000000, __pyx_v_sort_indexer=<unavailable>, __pyx_v_qs=__Pyx_memviewslice @ 0x0000600002f74490, __pyx_v_interpolation=0x0000000000000000) at groupby.c:40963 [opt]
   40960          __Pyx_XDECREF(__pyx_v_inter_methods);
   40961          __Pyx_XDECREF(__pyx_gb_6pandas_5_libs_7groupby_14group_quantile_11generator3);
   40962          __PYX_XCLEAR_MEMVIEW(&__pyx_v_sort_indexer, 1);
-> 40963          __PYX_XCLEAR_MEMVIEW(&__pyx_cur_scope->__pyx_v_qs, 1);
   40964          __Pyx_DECREF((PyObject *)__pyx_cur_scope);
   40965          __Pyx_XGIVEREF(__pyx_r);
   40966          __Pyx_RefNannyFinishContext();
</Details>

It looks like something is going wrong in Cython. For reference, this is the Cython code that seems to be triggering the segfault.

```
    if any(not (0 <= q <= 1) for q in qs):
        wrong = [x for x in qs if not (0 <= x <= 1)][0]
        raise ValueError(
            f"Each 'q' must be between 0 and 1. Got '{wrong}' instead"
        )
```
where qs is defined as ``const float64_t[:] qs``.

cc @jbrockmendel for help on this one.
no bright ideas here, maybe @da-woods can make something of this
I'll have a closer look later.

~Do you activate refnanny yourself? It should be turned off on most user code (it's really only an internal testing mechanism), but it's possible that Pandas enables it for their own test suit?~

~The reason I ask is that ` __Pyx_RefNannyFinishContext` should be a no-op without refnanny so hopefully shouldn't be able to crash~

Ignore this... It's not the refnanny line that's producing the error
I can reproduce it with

```
# testpandasbug.pyx
def f(const double[:] qs):
    if any(not (0 <= q <= 1) for q in qs):
        wrong = [x for x in qs if not (0 <= x <= 1)][0]
        raise ValueError(
            f"Each 'q' must be between 0 and 1. Got '{wrong}' instead"
        )
```

build it with `cythonize -if testpandasbug.pyx`

and run with:

```
>>> import testpandasbug
>>> import numpy as np
>>> testpandasbug.f(np.array([1.,2.,3.0]))
Fatal Python error: Acquisition count is -1 (line 3824)
Python runtime state: initialized
Traceback (most recent call last):
  File "testpandasbug.pyx", line 4, in testpandasbug.f
    raise ValueError(
ValueError: Each 'q' must be between 0 and 1. Got '2.0' instead
Aborted (core dumped)
```

So it definitely looks like a Cython bug and one that's fairly easy to reproduce
The next lot of failures also looks to be a Cython bug I think https://github.com/cython/cython/issues/4668
@da-woods Do you know if Cython changed anything with the way its pickling classes in 3.0? If I pickle a class that's compiled with Cython 0.29.28, the checksums don't match when I read it in with the class compiled with Cython 3.0.
Here is the error.
```
>>> import pickle
>>> pickle.load(open("pandas/tests/io/data/legacy_pickle/1.5.0/mixin.pkl", "rb"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stringsource>", line 6, in pandas._libs.tslibs.period.__pyx_unpickle_PeriodMixin
_pickle.PickleError: Incompatible checksums (222419149 vs 0xda39a3e = ())
```
With Cython master branch + cython/cython#4669.
And this is the class 
https://github.com/pandas-dev/pandas/blob/48d515958d5805f0e62e34b7424097e5575089a8/pandas/_libs/tslibs/period.pyx#L1545-L1614

(Generated by running
```
>>> import pickle
>>> from pandas._libs.tslibs.period import PeriodMixin
>>> pickle.dump(PeriodMixin(), open("pandas/tests/io/data/legacy_pickle/1.5.0/mixin.pkl", "wb"))
```
with Cython 0.29.28
Thanks a bunch.
[This](https://github.com/cython/cython/commit/f00af64bcc2cef4c3fb1433ed670ab5f9970e2cd) would be my best guess (although I can look into it properly sometime later to make sure).

I don't know how strict Cython should be about maintaining pickle compatibility, but I wonder if there's anything useful to do to make them compatible again?
We can certainly find a way to accept old pickles in Cython, so we should. The problem with the old checksum is that it uses MD5, which is not always available.
Couple of nitpicks, no serious objections.  I share everyone else's discomfort with the sub/rsub duplication pattern.

Will adding this to the CI make things easier for the cython folks?
> Couple of nitpicks, no serious objections. I share everyone else's discomfort with the sub/rsub duplication pattern.
> 
> Will adding this to the CI make things easier for the cython folks?

I would prefer if this is merged soon, and remaining comments are taken care of in followups. I'm finding it difficult to keep this up to date with main.
I suspect it'd be useful for Cython if this was merged (once everyone's happy with it, of course). It did catch a number of legitimate bugs (so presumably might catch more in future...)
OK.  I think the only other people/person regularly touching this code is me, so the added maintenance burden pre-cy3 won't be that big a deal.  I'm conceptually on board.
thanks @lithomas1 very nice. if you can make an issue for the noted comment.