Repository: cupy/cupy

Issue:
`pip install` completely ignores existing source builds and installed dependencies 
I suspect this has to do with #4619. 

I am on the latest master, and now every time I call `pip install -v -e .` two things happens:
1. These packages keeps being reinstalled despite I already have them in my env: setuptools, wheel, Cython, fastrlock
2. All modules are re-cythonized and recompiled from scratch, despite they've been built and nothing has changed 

I will try to build in a fresh env to see if something is wrong with my current env. But it's better to be confirmed independently.

cc: @kmaehashi @emcastillo 


Discussion:
> I will try to build in a fresh env to see if something is wrong with my current env. But it's better to be confirmed independently.

Still no luck. But, I should have also mentioned that I am (as always) building and developing CuPy in Conda, so by a fresh env I meant a fresh Conda env, which makes me wonder if PEP 517 is incompatible with Conda and I think the answer is yes:
- https://github.com/conda/conda/issues/9133
- https://github.com/conda/conda-build/issues/3507
- https://github.com/conda/conda-build/issues/3919
@jakirkham Is my theory correct that `pyproject.toml` would be in conflict with Conda in some way?
OK removing `pyproject.toml` resolves both issues, so at least we know a workaround. 

@kmaehashi I am not sure if there's still time for changes before the release, but I think it's better to not stick to PEP 517. I wanna note that Conda is the recommended way to run Python packages on Summit and NERSC afaik, and I know at least for the latter [they recommend users to build CuPy in a fresh Conda env](https://docs-dev.nersc.gov/cgpu/software/python/#cupy), so there will likely be surprises...

I am almost certain this is due to bad interaction between PEP-517 compliance and Conda. I set up the system Python in the NVIDIA's devel docker (no pyenv or anything though), and everything runs fine.
Would try adding `--no-deps` to `pip` to keep from installing dependencies
@jakirkham That would only address the first issue unfortunately, not the second. And there are still likely side effects. See https://github.com/conda/conda-build/issues/3919 and https://github.com/conda-forge/cvxpy-feedstock/pull/31. Removing `pyproject.toml` was the patch applied there.
It's also suggested on Conda-Forge's Gitter channel: https://gitter.im/conda-forge/conda-forge.github.io?at=5ec2ffb22f83c139ef8cca73

There's a bunch of things conda-build does to configure pip. May be worth [trying these]( https://github.com/conda/conda-build/blob/8f1b3517fd0c816ec78b6dadf4a912b849ecd58a/conda_build/build.py#L2574 ). In particular I think `PIP_NO_BUILD_ISOLATION` may be relevant, but the others may be relevant as well



RIght, but my argument is to both users and developers this is an unexpected regression. A simple build command without such complexities like `pip install -v -e .` (or without `-e`) is always the advice we give (by "we" I mean on CuPy's own doc, as well as HPC-related events that I participated).
Sure I have no strong feelings about the resolution am just trying to supply info 🙂
Ah, yes, indeed I didn't know Conda-Build took care of this. Thanks for pointers John!
Ofc happy to help 😄

Am curious if NumPy has similar issues with installs given they are using [`pyproject.toml`]( https://github.com/numpy/numpy/blob/9437402f1d60778cd723e6119044ef0510ef17c0/pyproject.toml#L1 ). Or is there something they are doing that we may be missing? (Not that we need to figure that out now if we need to get a fix in. Just another idea worth exploring)
Looking into it now.
I don't see what magic the numpy and scipy feedstocks have employed, but I tested it locally and adding `--no-deps` doesn't even solve the first issue. It still pulls down the said dependencies.
I skimmed the pip code, seems pip always install requirements from `pyproject.toml`
https://github.com/pypa/pip/blob/baaf66f70d84cf3367d5b43fec16762ca9703810/src/pip/_internal/distributions/sdist.py#L61-L64
although requirements from `setup_requires` are installed only when it is missing in the current environment:
https://github.com/pypa/pip/blob/baaf66f70d84cf3367d5b43fec16762ca9703810/src/pip/_internal/distributions/sdist.py#L90-L95
> In particular I think `PIP_NO_BUILD_ISOLATION` may be relevant

I think @jakirkham is right on this. Once this is set, regardless of whether other `PIP_NO_*` env vars or `--no-deps` etc flags are set or not, `pip` falls back to the old behavior and all is good too. This is likely why numpy/scipy feedstocks are happy. 

In fact, I think setting it is equivalent to removing `pyproject.toml`: 
> --no-build-isolation
> Disable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if this option is used.

https://pip.pypa.io/en/stable/reference/pip_wheel/?highlight=PIP_NO_BUILD_ISOLATION#cmdoption-no-build-isolation
Thanks for checking Leo. It's good to have the confirmed 🙂
Thanks for your insight, John! I couldn't possibly figure this out myself...
tl;dr: Workarounds for falling back to the old behavior (prior to #4619):
- remove `pyproject.toml` manually and then run `pip install -v .` as usual
- set `PIP_NO_BUILD_ISOLATION=False` (note the inverted logic) and then run `pip install -v .` as usual
- `pip install -v --no-build-isolation .`

All are equivalent. Conda recipe builds are immune to the change as it's done internally for us.
> @kmaehashi I am not sure if there's still time for changes before the release, but I think it's better to not stick to PEP 517. I wanna note that Conda is the recommended way to run Python packages on Summit and NERSC afaik, and I know at least for the latter they recommend users to build CuPy in a fresh Conda env, so there will likely be surprises...

I think it's ok to remove `pyproject.toml` for the next release.

> RIght, but my argument is to both users and developers this is an unexpected regression. A simple build command without such complexities like pip install -v -e . (or without -e) is always the advice we give (by "we" I mean on CuPy's own doc, as well as HPC-related events that I participated).

I couldn't fully understand the point here, I think the impact to users is very limited (they need to install Cython/fastrlock even if they already have it in their environment when installing from sdsit) so I think this is a regression for developers (increased rebuild time).

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -24,8 +24,7 @@
 
 
 requirements = {
-    # setup_requires remains here for pip v18 or earlier.
-    # Keep in sync with pyproject.yaml.
+    # TODO(kmaehashi): migrate to pyproject.toml (see #4727, #4619)
     'setup': [
         'Cython>=0.28.0',
         'fastrlock>=0.5',


Test patch: