Specify build requirements in pyproject.toml (PEP 517)
State: closed

- [x] closes #25193
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Verified this using the Dockefile:

```
FROM alpine:3.8
RUN apk update
RUN apk add g++ libstdc++ python3-dev bash git
RUN pip3 install --upgrade pip

COPY dist/pandas-0.25.0.dev0+86.gba77fb900.tar.gz /
RUN ls -l /
RUN pip3 install -vvv /pandas-0.25.0.dev0+86.gba77fb900.tar.gz
```

Labels:
name: Build
description: Library building on various platforms
name: Regression
description: Functionality that used to work in a prior pandas version
name: Dependencies
description: Required and optional dependencies

Comments:
Hello @xhochy! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-05-12 21:02:14 UTC
@jorisvandenbossche I hope this works around the issues that were reported before. If you know of others that came with `pyproject.toml` please link me to them. I'll have a look then.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=h1) Report
> Merging [#25227](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/1d1b14c7ea7c3e497b66453cacbb8230b26cb6a6?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25227/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #25227   +/-   ##
=======================================
  Coverage   92.15%   92.15%           
=======================================
  Files         166      166           
  Lines       52294    52294           
=======================================
  Hits        48194    48194           
  Misses       4100     4100
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <ø> (ø)` | :arrow_up: |
| #single | `42.29% <ø> (+0.02%)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=footer). Last update [1d1b14c...6bc1d5e](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@b48d1ff`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25227/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master   #25227   +/-   ##
=========================================
  Coverage          ?   91.73%           
=========================================
  Files             ?      173           
  Lines             ?    52839           
  Branches          ?        0           
=========================================
  Hits              ?    48472           
  Misses            ?     4367           
  Partials          ?        0
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.29% <100%> (?)` | |
| #single | `41.72% <100%> (?)` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/compat/numpy/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/25227/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9fX2luaXRfXy5weQ==) | `93.1% <100%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=footer). Last update [b48d1ff...cec7053](https://codecov.io/gh/pandas-dev/pandas/pull/25227?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> Yes, this is a known issue with NumPy and was fixed in the 1.13 series. I have now pinned the Python 3.6 build also to 1.13.1 and added an explanatory comment. As Alpine+Python3.6 was the initial origin for tackling the `pyproject.toml` again, I made this change.

This might ensure it works on alpine+python3.6, but then can give errors on other platforms in case you would eg do `pip install pandas numpy=1.12 --no-binary :all:` (or without the no-binary on platforms that don't have wheels, but where numpy 1.12 works).

I am not sure what the best way to go is here (although I am inclined to keep it at the lowest we support (1.12), so it works when numpy 1.12 works on the specific platform). 
But, it is a problem that many other packages also will encounter when adding (back) pyproject.toml file, like scipy, scikit-learn, ... So it might be interesting to know what they would do / to be consistent (as often those packages will be installed together)

cc @rgommers (since you were involved in the pyproject.toml file in scipy)



Pip support should finally be good enough now with pip 19, we're adding it back in SciPy soon: https://github.com/scipy/scipy/issues/8734

> (although I am inclined to keep it at the lowest we support (1.12), so it works when numpy 1.12 works on the specific platform).

Yes, this is the only correct way to do it, you have to build against the lowest supported numpy version for each Python version. If that lowest version varies (e.g. Python 3.7 + numpy 1.12.x won't work), then specify multiple versions like so: https://github.com/PyWavelets/pywt/commit/5e6d53a75b5c682ef1b8e40cefc57df7e7e2b399#diff-522adf759addbd3b193c74ca85243f7d
> Yes, this is the only correct way to do it, you have to build against the lowest supported numpy version for each Python version.

Yes, that's what we are doing (and did before as well before we removed the pyproject.toml again .. just like scipy did as well :)). 
But the question was more specifically about the situation for alpine linux, where apparently, numpy 1.12 doesn't work for python 3.6. So pinning to numpy 1.12, will mean that people on alpine (where typically no wheels are available so this pyproject.toml is of relevance) will still need to do a `--no-build-isolation` or `--no-use-pep517` to be able to install pandas. 
But I suppose there is not really a way around that?
maybe soln is to bump numpy requirement to 1.13? or higher 
Is is really that common that people are using Pandas with a numpy version older than a year? Updating numpy has been quite smooth in the last years, so I would have expected that this is one of the simpler migrations.
agree if bumping helps here let’s do it
only downside is that then this patch must be for 0.25 but that should be soonish anyhow
> only downside is that then this patch must be for 0.25 but that should be soonish anyhow

I would keep this patch for 0.25 anyway. We don't want to experiment with pyproject.toml in a bug fix release (it should not give that much troubles as last time we added it, but still)
> But I suppose there is not really a way around that?

Indeed there isn't. There are no features of `pyproject.toml` that allow you to make the numpy version that you build against depend on Alpine/MUSL vs other linux distros.

> Is is really that common that people are using Pandas with a numpy version older than a year?

Yes it is. I'm quite sure that that's more common than people using non-manylinux-compatible distros.

> maybe soln is to bump numpy requirement to 1.13? or higher

In this case, SciPy is anyway bumping to 1.13.3 as minimum numpy version for the next release. So bumping won't impact many users. So +1 for that solution.
The `pip` version on Azure is not yet `19.0.2`. I wonder whether it would be better to have the workaround in place to support the missing current directory in the `sys.path` rather than updating `pip` in the CI jobs.
Does this fully close 
https://github.com/pandas-dev/pandas/issues/25193, or is there an additional issue there with `cythonize` being called unconditionally?
If we are fine with cython being a build dependency (also for sdists, as pyproject.toml cannot really distinguish from source vs from sdist builds), then the unconditional call to cythonize should not really be a problem. 

But as I said already before, I don't think this PR closes #25193 *for 0.24.2* (we don't want to start using pyproject.toml in a bug fix release, and also the numpy version increase should go in 0.25).
>  I wonder whether it would be better to have the workaround in place to support the missing current directory in the `sys.path` rather than updating `pip` in the CI jobs.

It's not that urgent to merge this I think, so we might still be able to remove it before that. 
But, that said, we should maybe also think about adding `versioneer` to the build-system `requires`, instead of relying on the one included in the source, if we want to prevent that we need to have the current directory on the python path.
Bumping numpy comes with a bit more complexity (esp. re: compat code) - I've started a PR dedicated to just that in #25554.
@h-vetinari Thanks for taking care of the NumPy bump. I hope I can spend some time on this again once the bump is merged.
Yes, it is a good idea to do the numpy version bump in a separate PR.
> @xhochy: @h-vetinari Thanks for taking care of the NumPy bump. I hope I can spend some time on this again once the bump is merged.

The numpy bump has been merged a few days ago. :)
@xhochy can you merge master. we are now in 1.13 min, so you can remove a lot of these changes.
can you merge master
https://travis-ci.org/pandas-dev/pandas/jobs/529095120

```
ERROR: Error installing 'file:///home/travis/build/pandas-dev/pandas': editable mode is not supported for pyproject.toml-style projects. pip is processing this project as pyproject.toml-style because it has a pyproject.toml file. Since the project has a setup.py and the pyproject.toml has no "build-backend" key for the "build_system" value, you may pass --no-use-pep517 to opt out of pyproject.toml-style processing. See PEP 517 for details on pyproject.toml-style projects.
The command "ci/setup_env.sh" failed and exited with 1 during .
```

seems like https://github.com/pypa/pip/issues/6434
Yes, there was indeed a regression with the latest pip 19.1.0 for projects using editable installs and already having pyproject.toml file (although I think pip 19.1.1 which was just released should mostly fix that)

We could choose to postpone adopting a pyproject.toml somewhat more until the dust settles around it (I think people are working on a PEP update to include editable installs in PEP517/518, but that will take some more time). 
Is this a blocker for 0.25.0, or were all the blocking issue resolved elsewhere?
I don't think this is urgent (eg https://github.com/pandas-dev/pandas/issues/25193 which triggered, you already solved that on master)
closing; we need to see if we actually need this.