BLD: add option to specify numpy header location
State: closed

In some cases the numpy module might not be usable during build-time, especially when cross-compiling. (E.g. when compiling for arm32 on a x86-64 machine, the arm32 module is not usable at build time).

This makes meson fail, as it isn't able to figure out the location of numpy headers.

To allow an alternative way to find these headers, introduce a meson build option, where the location of the numpy headers can be specified.

In case numpy module cannot be loaded for some reason to query the include folder location, fall back to the value of this meson option.

- [x] closes #55305 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
I don't think the build failures showing now are related to this change, but rather a longstanding issue where build warnings were being suppressed with our Meson 1.2.1 pin. https://github.com/pandas-dev/pandas/pull/60681 should have the more comprehensive fix for that

For now, you can just remove any setup calls with `-Csetup-args="--werror"` 
Sure. Though there are still at least 1 or 2 that were broken by this - though if the list would get a bit cleaner, that wouldn't be a problem of course.
For the pre-commit error I think you just need to run `./scripts/generate_pip_deps_from_conda.py` and commit the reuslt.

I thought the pre-commit hook would do this for you, so maybe it was skipped. If that doesn't work can help take a closer look
You are right, that did the trick. But I'm not really sure why the wheel* workflows weren't kicked off
I see the problem - it's about the `PKG_CONF_PATH` env var, which should be visible for each process, that is building pandas (or wants to find numpy via meson for any other reasons). But github actions is fighting against me. I will experiment how to pass env vars back and forth between actions and workflows, but in a test repo, and hopefully will be back.
Ah yes, `cibuildwheel`-within-GHA is odd layering. I think this does what you want:

https://github.com/scipy/scipy/blob/689ebb9d0332b025d604b755febb68fb0d017a40/.github/workflows/wheels.yml#L125
I dunno. I tried it, but i doesn't seem to have any effect. It looks the environment variables are passed correctly, but they disappear in thin air by the time meson is executed. At this time I suspect that might be running into the same issue as https://github.com/mesonbuild/meson-python/discussions/604 - and it would need a clean build folder (but today I'm not brave enough to do `rm -rf` on random build machines), but that's only a desperate last thought, and I'm just afraid to accept the fact that it is cursed.

If you have any ideas, suggestions, please go for it, I'm all ears. Personally I have to put this to rest on my end for at least a day.
I might have missed a prior conversation but does NumPy definitely distribute the pkg-config file when installed via conda? Checking my local machine in an environment with 1.26.4 there is no directory for `<site_packages>/numpy/_core/lib`
When using meson's numpy resolution, there are two ways one can go in my understanding:

1. Use pkg-config. NumPy uses a non-standard location for the .pc file, I guess mostly because there is no standard location for .pc files provided by Python modules. This needs the PKG_CONFIG_PATH to be set to this custom path, otherwise pkg-config can't find it. I *believe* that there is always a numpy.pc file, though maybe depending on the version, the location might be different? (thinking this because of your comment) That unfortunately wouldn't make things easier. On the other hand I'm not too familiar with NumPy, I'm just trying to make it cross-compile - if you tell me that this is not right, I will believe it without second guessing.
2. Use /usr/bin/numpy-config. For this, meson needs to know where this executable/script is - it could be added to a meson*native file as a binary which that could be passed to meson (at least in some workflows this file seems to exist already, though I didn't check where it is coming from exactly), which would make the pkg-config method superfluous.

I do realize that currently it looks like a hack, and in the past days I only wanted to see it work at least once, to have at least a baseline - I suspect that half of the env-declarations are useless, and could be removed. But unfortunately I still have no idea why it doesn't work - and unless some miracle happens and I have some revelation, I think I might have to give this up - unfortunately I am not able to learn the ins and outs of a non-trivial CI setup due to time constraints :(
> in an environment with 1.26.4 there is no directory 

It's new in 2.0.0
@OldManYellsAtCloud thank you for the effort, this was helpful to understand the limitations of the `.pc` file. This PR is largely still in the right direction. Let's see if we can finish it off. @WillAyd do you want to take a stab at it, or do you want me to?
I don't have the time personally to work on this at the moment, but am happy to review any changes that come through. I do think this is close