BLD: Setup meson builds
State: closed

- [x] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

~~This can be merged before or after #49114. The meson build system is not going to work properly, anyways, until the PR enabling it as the default for PEP517 builds is merged anyways.~~

pandas can be compiled with meson now. Building pandas via pyproject.toml will be implemented in the future.


Labels:
name: Build
description: Library building on various platforms
name: Docs

Comments:
> Looks pretty good. Do we have to make changes to setup.py and/or CI still?

Sorry, it looks like I updated the title too eagerly. It looks like the pyproject.toml changes will have to go in with CI changes.

This would turn on meson builds for anyone doing a pip install from main. This is probably fine, since the meson builds pass all the tests anyways.

I don't think there will be any changes needed for setup.py(we'll keep it as a fallback, until we're ready to adopt meson).  
Is it a lot of effort to make this work for the development process as well? I think I'd prefer we use it consistently all the way through rather than just on install, as there are likely some build differences versus setuptools that we should catch in CI
There's a couple options here:
- merge meson.build, developers can use something like SciPy's `dev.py` to build, install with Meson so that it is importable, and test it, but it cannot be installed with pip or provide pip metadata
- update pyproject.toml to use meson, and only meson, affects pip install, people can still invoke `python setup.py bdist_wheel` if they want to build with setuptools
- delete setup.py, use only meson


Option 2 is the sensible approach for fallbacks, when porting to a new build system. Critically, it relies on setup.py natively supporting this, because...

... pip doesn't allow you to select a backend. Pip doesn't even allow you to ignore pyproject.toml and try the default setuptools (which it *would* do if you simply delete pyproject.toml before running pip). To add to the awkwardness, setuptools is busy deprecating and hoping to *remove* all support for a CLI, and the ecosystem around PyPA, pip, and PEP 517 encourages build backends to not provide a frontend, but use tools such as pip or build, which run whatever backend is in pyproject.toml.

Supporting multiple build systems during a migration is not something the python ecosystem has tried to take into account, and it cannot be generically expected to work. You have to pray that the build system ignores the hint to rely on pip, and provides its own frontend. Or you could edit pyproject.toml, either by hand or with a small python script, every time you want to switch which one you are trying.
Thanks for the input @eli-schwartz . Still a bit unclear to me but is the issue with doing the Option 3 of "only meson" that it doesn't work when doing a pip install? 
Option 3 means you better hope no one reports a bug with the port, because you cannot just tell them to "keep using setuptools for now, and we'll make sure to fix it in the next release". :D
If an issue pops up we can just pin or exclude certain versions of meson no? I think there are a lot of disadvantages to maintaining both setuptools and meson as a build backend from a code complexity perspective
> If an issue pops up we can just pin or exclude certain versions of meson no? I think there are a lot of disadvantages to maintaining both setuptools and meson as a build backend from a code complexity perspective

I think the main problem is that meson/meson-python hasn't cut a release with the necessary fixes to build pandas properly. 

As a side note, There is also a nasty issue affecting rebuilds of pandas where the pxi dependencies are not generated(mesonbuild/meson#8961). I have an idea in the works for this though(@eli-schwartz can you comment maybe on my proposed fixes?)

You can basically develop pandas using meson right now though. If you're on my meson-poc branch, you would just need to ``pip install git+https://github.com/mesonbuild/meson.git@master`` and ``pip install git+https://github.com/mesonbuild/meson-python.git@main``. Then you can do ``pip install --no-build-isolation . `` (editable installs won't work, since pandas is not built inplace anymore)

If you want to run tests, it would be ``pytest pandas ... --import-mode=importlib``. (This is necessary, since Python will try to look in the current directory for pandas, and it won't find the inplace extension modules(again meson doesn't build inplace) and will fail).
FWIW, I'm not too worried about maintaining the setuptools config. It doesn't really change much, and I guess the only issue would be if someone were to rename/remove/add a Cython module.
> I think the main problem is that meson/meson-python hasn't cut a release with the necessary fixes to build pandas properly.

Do you know how far out this is? I think should just wait until that becomes possible



> Then you can do `pip install --no-build-isolation . ` (editable installs won't work, since pandas is not built inplace anymore)

Is this always going to be a limitation?
> Do you know how far out this is? I think should just wait until that becomes possible

I can answer for the Meson part. We're going to start rc1 for the next version of Meson pretty soon. The release manager was hoping to do it today, but there was a testsuite failure that was only present in Debian packaging, and not in CI (a test case assumed there was a network available) so that needs to be fixed first.

> Is this always going to be a limitation?

There is an open ticket for meson-python to implement PEP 660 editable installs: https://github.com/mesonbuild/meson-python/issues/47

AFAIK the only holdup is that no one has put in the work.
Looking great so far. Are there no updates to CI required?
> Looking great so far. Are there no updates to CI required?

I'll open the next PR(for CI updates) as soon as this one's merged. Thanks for the reviews so far everyone!
I think it would be easier to just do all at once, i.e. have the documentation and CI updates to go along with this. Hard to know if this works / what to do as a developer otherwise
OK, this should be mostly green now. 

I think the 32-bit is the last tough one to figure out. Is it OK if we keep using setuptools for that one? I'm not too thrilled about putting that much effort into a platform we're going to drop soon anyways.
Docstrings probably requires just cd'ing into the directory.
Is it just individual test errors for 32 bit platforms? I recall the same issue with CMake and ended up just changing some of the tests. Might be helpful to reference #47380
> Congrats on green!

Thanks. Sorry if I was spamming you with push notifications. CI was very tough to wrangle :(. 
> Looking good. Some comments - I think we should wait for the new meson / meson-python releases to merge?

Well meson 0.64.0 is already out and meson 0.65.0 is ~3 months away. Unfortunately, I'm not sure if any fixes are backportable

On the meson-python side, my fix for the sdist has been merged, but there are issues with the version in the wheel name being off, as well as some unnecessary files included in the wheel. Those are pretty easy issues to workaround, though, and a fix shouldn't be too hard either. 

Honestly, I think it's fine merging as is. I'e never had any issues with meson/meson-python master/main breaking builds over the last ~3 months working on this. 

The problem with letting this sit, is that merge conflicts/more broken tests will start popping up, and I don't think I have the energy to keep this open any longer than a couple weeks. 
> The problem with letting this sit, is that merge conflicts/more broken tests will start popping up, and I don't think I have the energy to keep this open any longer than a couple weeks.

I've had that same experience with SciPy. Worse even is if someone tweak `setup.py` files in a way that doesn't result in immediate breakage - then you are silently missing compile settings etc.
Gave this a few tries locally and ran great. What is the suggested developer workflow with this now? 
> Gave this a few tries locally and ran great. What is the suggested developer workflow with this now?

#49683 
@pandas-dev/pandas-core if anyone can try this out and give feedback would be great
@lithomas1 I'd probably prefer to have some if not all the documentation included as part of this. That way developers have a reference on what to look at to use this.

+/- 0 on keeping this side by side with setup.py. Obviously some risk to having to duplicate work, but we don't change the build system that often. Probably OK to have both for a bit

I think some open questions on my end would be how to enable the debug build (worth having in the extension debugging documentation) as well as how to control warnings / build verbosity in CI as needed
> @lithomas1 I'd probably prefer to have some if not all the documentation included as part of this. That way developers have a reference on what to look at to use this.
> 
> +/- 0 on keeping this side by side with setup.py. Obviously some risk to having to duplicate work, but we don't change the build system that often. Probably OK to have both for a bit
> 
> I think some open questions on my end would be how to enable the debug build (worth having in the extension debugging documentation) as well as how to control warnings / build verbosity in CI as needed

I have a strong preference here.  I'd like to test this out in a Windows environment (where I do my pandas work).  @lithomas1 can you include the docs on what people have to do in order to use meson on their local machines?
> > @lithomas1 I'd probably prefer to have some if not all the documentation included as part of this. That way developers have a reference on what to look at to use this.
> > +/- 0 on keeping this side by side with setup.py. Obviously some risk to having to duplicate work, but we don't change the build system that often. Probably OK to have both for a bit
> > I think some open questions on my end would be how to enable the debug build (worth having in the extension debugging documentation) as well as how to control warnings / build verbosity in CI as needed
> 
> I have a strong preference here. I'd like to test this out in a Windows environment (where I do my pandas work). @lithomas1 can you include the docs on what people have to do in order to use meson on their local machines?

Thanks for offering to try it. Docs have been updated.

I also wrote up #49683, which has everything in one place. 
Hey @lithomas1 

I tried this out, in a WSL2 Ubuntu environment, here's what I ran:
```
 2021  python checkout-pr.py "git pull git@github.com:lithomas1/pandas.git add-meson-build-files" pull
 2022  git clean -fxfd pandas/
 2023  mamba env remove -n pandas-dev
 2024  mamba env create
 2025  mamba activate pandas-dev
 2026  pip install . --no-build-isolation --config-settings="builddir=builddir"
 2027  pytest pandas/tests/io/formats/test_info.py --import-mode=importlib
```

and it seems to have worked perfectly fine

Well done! 🥳 🎉 

---

(note: [checkout-pr.py](https://gist.github.com/MarcoGorelli/ed6249f88644d89254ec1bf5f85405cc#file-checkout-pr-py) is a convenience script I have for checking out PRs)
@Dr-Irv were you able to get this working on windows?
> @Dr-Irv were you able to get this working on windows?

Was on vacation last week - still catching up, but this is on my list of things to do!
Any other feedback here? Lots of great work want to be cognizant of effort to constantly maintain PR
@pandas-dev/pandas-core giving this a week for more feedback, but if no outstanding objections plan to merge Wed Dec 28 to keep this moving along. Should sit side-by-side with setuptools for a bit. More iterations / cleanup to come later but excited for all the work done on this
I have tried this out on Windows, and frankly am confused.  The new docs say:
```text
# Build and install pandas
python setup.py build_ext -j 4
python -m pip install -e . --no-build-isolation --no-use-pep517

# Alternatively, if you would like to try out our new build system
# based on meson
pip install . --no-build-isolation --config-settings="builddir=builddir"
```

But you still have to do `python setup.py build_ext -j 4`. to build the extensions. The two alternatives are
1.  `python -m pip install -e . --no-build-isolation --no-use-pep517`
2.  `pip install . --no-build-isolation --config-settings="builddir=builddir"`

So I don't see what the "new build system based on meson" has provided beyond a different way of installing the local pandas.  I thought this new build system would fix the issue creating collisions when trying to do parallel compilations that I reported in #30873 .  But I still had to do the `setup.py build_ext` step to build the C extensions.  How is `meson` invoked ?

Am I missing something, or are the docs incomplete?