Re-land PyCapsule
State: closed

Looks like this was accidentally reverted https://github.com/pandas-dev/pandas/pull/51952#issuecomment-1469076603 https://github.com/pandas-dev/pandas/pull/51951#issuecomment-1470386562

Labels:
name: Build
description: Library building on various platforms

Comments:
"accident" is a nice word for it.  i jumped the gun.  my bad.
I renamed this, to avoid confusion.

I've also changed my mind on backporting. 
Let's try this for 2.0, otherwise other C/Cython changes can be very hard to backport.
Are we sure that this isn't breaking anything?
> Are we sure that this isn't breaking anything?

This is the documented way to share C functions across extension modules. See https://docs.python.org/3/extending/extending.html#using-capsules
That's not what I meant. This is a pretty big pr where it's easy to miss something
The diff is a bit deceiving. The only real large addition to the code base is the 2 new include files. Everything else is just moving things to a better home (ex: function definitions of out parse_helper.h into pd_parser.c).

This actually improves type safety; the parsers.pyx changes before were happy redeclaring functions that are really implemented as returning a double * to functions returning a float64_t *. Guessing Cython relied on letting the linker resolve symbols before, but that returns a compilation warning/error in the current design.
Hmm why would this make backporting Cython changes any different?
The only file git doesn't correctly recognize as being moved is parse_helper.h which was last changed in 2020
Actually, looking at this again, I guess the only case backports would be affected are if a function was added/changed.

IIUC, the new process with the PyCapsule, is to modify the function in the original file, change the declarations in ``pd_parser``/``pd_datetime``, and then modify pyx imports. Is that right?

I guess leaving it in only 2.1 could be fine, but wasn't sure if you were planning on refactoring more (in which case I'd expect this backported).

In general, I like to keep CI/build stuff as close as possible between the main/release branch.
The only things that I think would make an issue with a backport are:

  1. Updating parse_helper.h
  2. Adding a new C datetime function
  3. Changing the declaration of a C datetime function

I don't think 1. is likely to happen and 2. And 3. aren't really things we should backport

> IIUC, the new process with the PyCapsule, is to modify the function in the original file, change the declarations in `pd_parser`/`pd_datetime`, and then modify pyx imports. Is that right?

Generally, if we:

  1. Update a function within a module (ex: np_datetime.c) then that's it; the capsule changes nothing
  2. New/changed function declaration; would need to be reflected in the capsule as well (think of the capsule as the internal API)
  3. Decide we want to use the capsule functions in a Cython module that hasn't previously used the capsule; we need to cimport the `import_pandas_X` function and invoke it; this is the same functionality we already have with `cnp.import_array()` and `PyDateTime_IMPORT`


Since it appears this is a reorganization, I suppose this could go in 2.0, but no strong preference on my end. Would you still prefer this go into 2.1 @lithomas1? 
No strong preference anymore, let's just put in in main for now, and see if it sticks this time.