BLD: Add pyproject.toml
State: closed

* Adds pyproject.toml to support this.

* Bumps minimum Cython version to the version supporting Python 3.8.

Closes https://github.com/pandas-dev/pandas/issues/20775
xref https://github.com/pandas-dev/pandas/issues/27435 (check if closes)
xref https://github.com/pandas-dev/pandas/pull/25227, https://github.com/pandas-dev/pandas/pull/20718, https://github.com/pandas-dev/pandas/pull/16745

Labels:
name: Build
description: Library building on various platforms

Comments:
I take it there's no way to get the pyroject.toml stuff in setup.cfg?
should/could we test for the desired behavior?  or rare enough to check manually?
> I take it there's no way to get the pyroject.toml stuff in setup.cfg?

Nope.

> should/could we test for the desired behavior? or rare enough to check
manually?

We could use pip to compile a wheel from source and install that wheel into
our test environment? Dunno how much effort that will be.

On Tue, Sep 10, 2019 at 2:26 PM jbrockmendel <notifications@github.com>
wrote:

> should/could we test for the desired behavior? or rare enough to check
> manually?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28374?email_source=notifications&email_token=AAKAOIVAHBUNQ263Z2QEXGLQI7YIDA5CNFSM4IVLHR22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6MHDUA#issuecomment-530084304>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVN5KUF3ZVEUJ2CTCLQI7YIDANCNFSM4IVLHR2Q>
> .
>

I'm having trouble with editable installs. I need to pass the `--no-build-isolation` in order to use the extensions built with `python setup.py build_ext -i`.
So from what I can tell, `pip install -e .` isn't going to play nicely with a previous `build_ext --inplace` (https://discuss.python.org/t/pip-19-1-and-installing-in-editable-mode-with-pyproject-toml/1553/68). So I guess the recommendation is to use `python setup.py develop` rather than `python -m pip install -e .`

Has anyone else followed this setup tools / pip issue? Does that sound right?
The editable install is a known issue (for me that is a reason to not yet use pyproject.toml on other projects, but for pandas the benefits are probably bigger than the problem with `pip install -e .`. 
Pip install will do build isolation once there is a pyproject.toml, which is something you exactly don't want (IMO) when doing an editable install. 
So as you say either passing a `--no-build-isolation`, or either `python setup.py develop`.

Can we do adding the pyproject.toml separately from the stop shipping cythonized files? Eg a separate PR as precursor for this (given all the previous problems I prefer to do this very explicitly and ) Or at least not hide it in a commit / PR that seems to be about something else. 
> So as you say either passing a --no-build-isolation, or either python setup.py develop.

From a comment that I made earlier when running into this (https://issues.apache.org/jira/browse/ARROW-5210), it seems that I needed to do `pip install -e . --no-use-pep517 --no-build-isolation`. But I don't remember from the top of my head why both flags were needed.
> Can we do adding the pyproject.toml separately from the stop shipping cythonized files? Eg a separate PR as precursor for this (given all the previous problems I prefer to do this very explicitly and ) Or at least not hide it in a commit / PR that seems to be about something else.

Sure. I think Pyproject.toml should be first, and needs to be back ported. I'll do that here, since it's dominated the discussion.
Windows people, is it possible to do a glob that expands to a file in a `.bat` file? E.g. I have 

```python
bash-5.0$ ls wheelhouse
pandas-0.25.0+323.g55294ca9a.dirty-cp37-cp37m-macosx_10_14_x86_64.whl
bash-5.0$ pip install wheelhouse/*.whl
```

That puts the single `whl` file in the `pip install` command.
Just the SSL certificate doc failure.

Edit: though this should likely wait until https://github.com/pandas-dev/pandas/pull/28391 is in.
@TomAugspurger it might be that the `--no-use-pep517` is not needed, as they reverted some things in a pip bug fix release: https://pip.pypa.io/en/stable/news/#id50
Huh, I'll try again, but I recall needing it and I'm using pip 19.2.3 (that change was in 19.1.1)
lgtm. merge when ready.
> it might be that the --no-use-pep517 is not needed

Oh, I understand now. The `--no-use-pep517` is indeed not necessary. The `--no-build-isolation` *is* necessary to reuse previously built extensions.
All green.

As a reminder, I'll followup with a PR to clean up our sdist (including removing the cythonized files).
Oh, and just to make sure everyone is aware, @pandas-dev/pandas-core the proper way to rebuild extensions is now

```python
python setup.py build_ext -i  # -j 8 for parrallel
python -m pip install -e . --no-build-isolation
```

if you leave off the `--no-build-isolation` you'll be rebuilding from scratch each time.
@TomAugspurger just to be clear we have to run the `pip install -e` locally now when rebuilding instead of just `python setup.py build_ext --inplace`?
I second Will's question.  My usual workflow involves lots of `git worktree` and very little virtualenv/condaenv
If you weren't previously using a local editable install then there's no change to your workflow.
Sounds good. We should probably update the contributing guide for that - I'll open an issue I think community will pick up
Nvm...missed already here - thanks for that