Investigate using abi3 wheels (Stable ABI + limited C API)

Copying and pasting relevant info from https://github.com/scipy/scipy/issues/23791 (thanks @rgommers for the initial write-up and super detailed info).

It looks like this could "just work" for us so worth a try. The attraction of that is that it will allow building abi3 wheels, meaning we need one (or two for now, with free-threading) wheel per platform rather than N per platform with N the number of supported Python versions.

### scikit-learn specific questions

- Python 3.11 will be supported one more year so we could do one Python 3.11 wheel and one Python >= 3.12 wheel
- I guess we would build one Python >= 3.12 wheel but still tests all the Python versions. Testing on only one Python version seems too brittle, you could imagine introducing Python 3.13 say issue and not realize until the release. If we test all the Python version that's not going to reduce the CI time that much (most of the time is spent in tests not in build). Not sure you can build once and test for all the Python versions easily with cibuildwheel. At worst you build many times the same wheel, since this is not where most of the CI time is spent.
- other things I have forgotten

### Details


- CPython: the Stable ABI has been taken more serious over the past couple of years and gaps have been filled. Python 3.12 seems to be the first version that has everything that extension module generators need. And Python 3.12 will be our lowest supported version very soon.
- NumPy: NumPy headers should have complete guards since 2.0.0 so extension modules that use NumPy headers can use the limited C API. There are a few things in the NumPy C API that are then unavailable (e.g., [`PyArray_DTypeMeta](https://numpy.org/devdocs/reference/c-api/types-and-structures.html#c.PyArray_DTypeMeta)), but I don't think we're using those objects in SciPy directly. Support isn't super well tested, but is likely to work as is for SciPy.
- Meson: supported since 1.3.0, see limited_api in [mesonbuild.com/Python-module.html#extension_module](https://mesonbuild.com/Python-module.html#extension_module)
- meson-python: supported since 0.14.0, see [the meson-python.limited-api build config option](https://mesonbuild.com/meson-python/reference/pyproject-settings.html#cmdoption-arg-tool.meson-python.limited-api)
Cython: Supported fairly well in 3.1.x, and more improvements landing regularly. See docs at [cython.readthedocs.io/en/latest/src/userguide/limited_api.html](https://cython.readthedocs.io/en/latest/src/userguide/limited_api.html)

**Performance impact should be measured, but is expected to be modest.** [This section of the Cython docs](https://cython.readthedocs.io/en/latest/src/userguide/limited_api.html#performance) discuss the expected performance impact based on type of usage: "Where the majority of the work involves C-level code the performance loss is likely to be low. This includes code that makes heavy use of typed memoryviews, or code that mainly calls an external C library. Where the majority of the work involves interacting with Python objects the cost is likely to be more significant." SciPy's usage is of the former kind, we don't deal with lots of small objects. So it's probably in the 0% - 10% range, with the median being 0% (i.e., it doesn't matter at all for most functions written in Cython).

**How to enable building against the limited C API:** this can be done globally, per extension, or per type of binding generator. E.g., for Cython, it only requires adding `-DPy_LIMITED_API=0x030C0000` as a flag to `cython_c_args` in `scipy/meson.build` to get going. Some more plumbing is involved to ensure the file extension contains `.abi3`, but that can be left to the very end.

COMMENT:
> * Not sure you can build once and test for all the Python versions easily with cibuildwheel.

You can (I was just doing it for Cython itself this afternoon...)

Essentially you need to do all the wheels for a single platform with a single runner. When it iterates over the Python versions it will magically work out that you've already built a compatible stable ABI wheel. It will then decide not to rebuild the wheel, but it will rerun the tests.

Labels: Build / CI