BLD, CI: Use `cibuildwheel` to build Emscripten/Pyodide wheels, push nightlies to Anaconda.org
State: closed

- [x] This is a follow-up PR to #57896 as stated in #57891
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

## Description

This PR duplicates the Pyodide wheel builder job that was set up in #57896 into `wheels.yml`, except that in this case the wheel is not tested and the wheel builder uses `cibuildwheel` enabled with Pyodide to build the wheels. Subsequently, the `upload_wheels.sh` script will be used with the appropriate repository secret to upload the Pyodide wheel to https://anaconda.org/scientific-python-nightly-wheels/pandas.

## Additional context

1. This is how this was done for NumPy: https://github.com/numpy/numpy/pull/26134
2. At a later time, these wheels will be used to configure interactive documentation via JupyterLite through `jupyterlite-sphinx` that acts upon doctest-based examples. The blocker in that regard is setting up a method to pre-install these wheels to provide them to the `jupyterlite-pyodide-kernel` and make them available at run time, and progress is being made in this area.

Adding an xref for tracking purposes, the action from `main` will be used until a stable release of `cibuildwheel` is available that can build Pyodide wheels: https://github.com/pypa/cibuildwheel/pull/1456

Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
Looks like cibuildwheel might be adding pyodide here
https://github.com/pypa/cibuildwheel/pull/1456

Maybe we should wait for that PR?
Hi @lithomas1, I would advise holding off on that PR – it's stalled for many reasons, for example the fact that Pyodide does releases bi-annually with updated Python versions and Emscripten versions which will break the ABI – i.e., while the specification for breaking changes and ABI stability for the platform might exist, it has not been formalised through the means of a PEP yet; and because PyPI has not yet received support through a PEP for formalising the wasm32 tag through `packaging` and `pip`, which means that `micropip` remains the only choice for installing these wheels (which, then, does not come with `resolvelib`, AFAIK) and wheels need to be pushed to alternative indices – Anaconda.org, in this case. However, out-of-tree build support in the ecosystem is improving day by day; it could be a viable candidate sometime soon. In the meantime, this current method of building should be a reasonable alternative since it's tightly integrated into and similar to Pyodide's in-tree builds with the `pyodide build` CLI.
I ended up asking around on that PR and it has been merged now after more than a year of effort; `cibuildwheel` hasn't had a release with it yet, but following these recent developments I have tried it for NumPy and it works nicely: https://github.com/numpy/numpy/pull/26570 – let me replace the configuration here as well and rebase this branch.

I would still recommend using this existing configuration because it's much more stable and it will allow us to get these wheels uploaded to Anaconda.org much quicker, but in any case, that's not a high-priority pressing issue right now, so I would not mind if there is a delay with that and I understand that the review process can take time, running on its own accord.
Both the wheel build and the tests pass 🎉: https://github.com/pandas-dev/pandas/actions/runs/9307939926/job/25620279093

I'll convert this to a draft for now – I will mark it as ready for review as soon as a stable `cibuildwheel` release hits.
Based on https://github.com/pandas-dev/pandas/actions/runs/9466987065/job/26080112499, I think there are some failing tests that are not running in the regular Emscripten job, and I am not sure why yet. I'll fix them as necessary in subsequent commits.
Ready for review, @mroeschke and @lithomas1.
I am -1 on adding this to CI or claiming any official support; this just makes a more Arrow-centric future even more difficult
@WillAyd, thanks for your comment. Could a reasonable alternative be to upload these nightly wheels silently without an announcement or a CHANGELOG entry, at least for now? It has been a while since the Pyodide job was added in #57896 and I figure that it has been running smoothly without any outstanding errors – I don't see anything new having turned up in issues or PRs with either "Emscripten" or "Pyodide" being referenced :)

Moreover, https://github.com/apache/arrow/pull/37822 just got merged a few days ago, and though https://github.com/pyodide/pyodide/issues/2933 isn't going to get into Pyodide 0.26.2, it will be going into Pyodide 0.27.X later in the year, marking official Arrow support in/for Pyodide. Though I understand and I am aware of the PDEP and the ramifications by now on bandwidth use and the drastic increase in download sizes, an Arrow-centric future might be reasonably possible and is definitely getting closer. I hope that you can reconsider your stance!
@agriyakhetarpal that's great news on the pyarrow support. With that being the case I have no concerns
@lithomas1, I did what you requested and the workflow runs on my fork successfully: https://github.com/agriyakhetarpal/pandas/actions/runs/9905625217/job/27365583191. @mroeschke, this may slightly contradict your previous suggestion in https://github.com/pandas-dev/pandas/pull/58647#discussion_r1672672765 about `cibuildwheel` taking care of future Python versions, because I have hardcoded the Python version (3.12) again here for the build identifier – however, `cibuildwheel` isn't going to bump the Pyodide version anytime soon, since there is considerable time with the semiannual release approach for Pyodide. Please feel free to tag me if anything breaks, because, as the one who has proposed this addition to this workflow, I intend to maintain it too and keep it updated. This is now ready for merging.
Thanks @agriyakhetarpal