Repository: python-poetry/poetry

Issue:
poetry install does not honor `build-system.requires`
<!--
  Hi there! Thank you for discovering and submitting an issue.

  Before you submit this; let's make sure of a few things.
  Please make sure the following boxes are ticked if they are correct.
  If not, please try and fulfill these first.
-->

<!-- Checked checkbox should look like this: [x] -->
- [ ] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option).

<!--
  Once those are done, if you're able to fill in the following list with your information,
  it'd be very helpful to whoever handles the issue.
-->

- **OS version and name**: Ubuntu 20.04.4
- **Poetry version**: 1.2.0b2
- **Link of a [Gist](https://gist.github.com/) with the contents of your pyproject.toml file**:
<details>
<summary>pyproject.toml</summary>

```
[tool.poetry]
name = "meow"
version = "0.1.0"
description = ""
authors = [""]
build = "build.py"

[tool.poetry.dependencies]
python = "^3.9"
SoundFile = "0.10.*"
crcmod = "^1.7"
librosa = ">=0.8.0"
numba = "^0.56.0"
protobuf = "3.14.0"
tensorboard = "^2.10.0"
torchaudio = {version = "=0.12.1+cu113", source="torch"}
torch = {version = "=1.12.1+cu113", source="torch"}

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu113"
secondary = true

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0", "setuptools>=40.6.0", "wheel", "numpy", "pybind11"]
build-backend = "poetry.core.masonry.api"
```

</details>

## Issue
<!-- Now feel free to write your issue, but please be descriptive! Thanks again 🙌 ❤️ -->
Running `poetry build` with the above `pyproject.toml` does the right thing:
```
Preparing build environment with build-system requirements poetry-core>=1.0.0, setuptools>=40.6.0, wheel, numpy, pybind11
[...snip...]
```

Running `poetry install` fails because it attempts to build the project without installing `build-system.requires` packages.

I would have expected the install command to create an isolated environment, install the build requirements and perform the build in that isolated environment, and then install the produced artifact in my current environment.


Discussion:
Same issue on macOS 12.5. A dirty hack is to add the following at the top of `build.py`:
```python
import subprocess

try:
    import pybind11
except ImportError:
    subprocess.run(["pip", "install", "pybind11"])
```

The exact same issue was described in #2789, which was marked as solved, but the issue is still present on Poetry 1.2.0b3. I didn't check if this is a regression from when [this PR](https://github.com/python-poetry/poetry/pull/5401) was merged.
This is currently intentional -- Poetry has never respected the `build-system` entry for a Poetry-managed project. We might want to establish consensus on this being a desired behavior, since currently you can install any needed build dependencies using a Poetry group or similar (but this will result in duplication of deps in pyproject.toml).
Re: this being a regression, the changes in that PR were explicitly scoped to use of build scripts.

Also, Poetry does not perform builds in an isolated/pep 517 environment, but instead uses the 'main' environment `poetry install` uses, as this is the only way to use lockfile versions during the build with the current architecture. It is possible to re-architect this so that a clean build environment is used and the `build-system` deps are only installed there (as opposed to polluting the `dev` group implicitly or similar) -- however that will require the implementation of a lockfile-aware `poetry-core` first (which we have already discussed/planned).
Is this the same issue as `[tool.poetry.dependencies]` deps being unavailable to `tool.poetry.build.script` in 1.2.0 despite being installed?
+1, having exact same issue `tool.poetry.build.script` is not using dependencies from `[tool.poetry.dependencies]`
@neersighted  For me this issue occurs when I have a GIT-dependency that has `jupyter-packaging` as a build dependency. (specifically this fork here: https://github.com/randolf-scholz/jupyterlab_templates)

Also, it does seem poetry uses an isolated environment to build this package.

```
Command ['/tmp/tmpx6y_j1ku/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output: 
Traceback (most recent call last):
  File "/home/rscholz/Projects/KIWI/python-venv/.venv/src/jupyterlab_templates/setup.py", line 4, in <module>
    from jupyter_packaging import (
ModuleNotFoundError: No module named 'jupyter_packaging'
```

<details> <summary> `/tmp/tmpx6y_j1ku/.venv/bin/pip list` </summary>

```
Package    Version
---------- -------
build      0.7.0
packaging  21.3
pep517     0.12.0
pip        22.3.1
pyparsing  3.0.9
setuptools 65.5.1
tomli      2.0.1
wheel      0.37.1
```

</details>


Your issue is entirely different @randolf-scholz -- this issue is about `build-system.requires` for Poetry-managed projects, and not for dependencies that don't use Poetry. Please reach out via Discord or Discussions for support.
I believe @randolf-scholz's issue is what I've reported as a separate issue in #7702, I couldn't find any other issue about it.
I raised this at https://github.com/python-poetry/poetry/issues/7909 as a potentially separate issue, but it might be the same thing I'm encountering.

The pyproject.toml is available on the linked issue, but for a TLDR here, we have a `build.py` that compiles cython, and so our build requires include both cython and numpy. A `poetry build` works fine. But a `poetry install` does not, because it seems when "Getting requirements to build editable",  `setuptools/build_meta.py` is run (via the `--isolated` tag in the parent pip invocation) in an environment which does not respect the build requirements.

Full stack from our failing `docker build` below:


```
=> [builder 11/12] RUN rm -f /app/id_rsa && poetry build                                                                                                           17.1s
 => ERROR [builder 12/12] RUN poetry install --only-root -vvv                                                                                                        3.7s
------
 > [builder 12/12] RUN poetry install --only-root -vvv:
#0 0.784 Loading configuration file /app/poetry.toml
#0 0.818 Skipping virtualenv creation, as specified in config file.
#0 0.998 Installing dependencies from lock file
#0 1.181 
#0 1.181 Finding the necessary packages for the current system
#0 1.195 
#0 1.195 Installing the current project: optimiser (0.1.0)
#0 1.195   - Building package optimiser in editable mode
#0 1.195   - Falling back on using a setup.py
#0 3.141 
#0 3.141   Stack trace:
#0 3.141 
#0 3.141   2  /opt/venv/lib/python3.9/site-packages/poetry/utils/env.py:1543 in _run
#0 3.206       1541│                 return subprocess.call(cmd, stderr=stderr, env=env, **kwargs)
#0 3.206       1542│             else:
#0 3.206     → 1543│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
#0 3.207       1544│         except CalledProcessError as e:
#0 3.207       1545│             raise EnvCommandError(e, input=input_)
#0 3.207 
#0 3.207   1  /usr/local/lib/python3.9/subprocess.py:424 in check_output
#0 3.261        422│         kwargs['input'] = empty
#0 3.261        423│ 
#0 3.261     →  424│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#0 3.261        425│                **kwargs).stdout
#0 3.261        426│ 
#0 3.261 
#0 3.261   CalledProcessError
#0 3.261 
#0 3.261   Command '['/opt/venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/opt/venv', '--upgrade', '--no-deps', '-e', '/app']' returned non-zero exit status 1.
#0 3.261 
#0 3.261   at /usr/local/lib/python3.9/subprocess.py:528 in run
#0 3.317        524│             # We don't call process.wait() as .__exit__ does that for us.
#0 3.317        525│             raise
#0 3.317        526│         retcode = process.poll()
#0 3.317        527│         if check and retcode:
#0 3.317     →  528│             raise CalledProcessError(retcode, process.args,
#0 3.317        529│                                      output=stdout, stderr=stderr)
#0 3.317        530│     return CompletedProcess(process.args, retcode, stdout, stderr)
#0 3.317        531│ 
#0 3.317        532│ 
#0 3.317 
#0 3.317 The following error occurred when trying to handle this error:
#0 3.317 
#0 3.317 
#0 3.317   Stack trace:
#0 3.317 
#0 3.317   2  /opt/venv/lib/python3.9/site-packages/poetry/utils/pip.py:56 in pip_install
#0 3.319        54│ 
#0 3.319        55│     try:
#0 3.319     →  56│         return environment.run_pip(*args)
#0 3.319        57│     except EnvCommandError as e:
#0 3.319        58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
#0 3.319 
#0 3.319   1  /opt/venv/lib/python3.9/site-packages/poetry/utils/env.py:1505 in run_pip
#0 3.379       1503│         pip = self.get_pip_command()
#0 3.379       1504│         cmd = pip + list(args)
#0 3.379     → 1505│         return self._run(cmd, **kwargs)
#0 3.379       1506│ 
#0 3.379       1507│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:
#0 3.379 
#0 3.379   EnvCommandError
#0 3.380 
#0 3.380   Command ['/opt/venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/opt/venv', '--upgrade', '--no-deps', '-e', '/app'] errored with the following return code 1
#0 3.380   
#0 3.380   Output:
#0 3.380   Obtaining file:///app
#0 3.380     Installing build dependencies: started
#0 3.380     Installing build dependencies: finished with status 'done'
#0 3.380     Checking if build backend supports build_editable: started
#0 3.380     Checking if build backend supports build_editable: finished with status 'done'
#0 3.380     Getting requirements to build editable: started
#0 3.380     Getting requirements to build editable: finished with status 'error'
#0 3.380     error: subprocess-exited-with-error
#0 3.380     
#0 3.380     × Getting requirements to build editable did not run successfully.
#0 3.380     │ exit code: 1
#0 3.380     ╰─> [21 lines of output]
#0 3.380         Traceback (most recent call last):
#0 3.380           File "/opt/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#0 3.380             main()
#0 3.380           File "/opt/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#0 3.380             json_out['return_val'] = hook(**hook_input['kwargs'])
#0 3.380           File "/opt/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
#0 3.380             return hook(config_settings)
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 450, in get_requires_for_build_editable
#0 3.380             return self.get_requires_for_build_wheel(config_settings)
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
#0 3.380             return self._get_build_requires(config_settings, requirements=['wheel'])
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
#0 3.380             self.run_setup()
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
#0 3.380             super(_BuildMetaLegacyBackend,
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
#0 3.380             exec(code, locals())
#0 3.380           File "<string>", line 48, in <module>
#0 3.380           File "/app/builder.py", line 3, in <module>
#0 3.380             import numpy as np
#0 3.380         ModuleNotFoundError: No module named 'numpy'
#0 3.380         [end of output]
#0 3.380     
#0 3.380     note: This error originates from a subprocess, and is likely not a problem with pip.
#0 3.380   error: subprocess-exited-with-error
#0 3.380   
#0 3.380   × Getting requirements to build editable did not run successfully.
#0 3.380   │ exit code: 1
#0 3.380   ╰─> See above for output.
#0 3.380   
#0 3.380   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 3.380   
#0 3.380 
#0 3.380   at /opt/venv/lib/python3.9/site-packages/poetry/utils/env.py:1545 in _run
#0 3.440       1541│                 return subprocess.call(cmd, stderr=stderr, env=env, **kwargs)
#0 3.440       1542│             else:
#0 3.440       1543│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
#0 3.440       1544│         except CalledProcessError as e:
#0 3.440     → 1545│             raise EnvCommandError(e, input=input_)
#0 3.440       1546│ 
#0 3.441       1547│         return decode(output)
#0 3.441       1548│ 
#0 3.441       1549│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
#0 3.441 
#0 3.441 The following error occurred when trying to handle this error:
#0 3.441 
#0 3.441 
#0 3.441   Stack trace:
#0 3.441 
#0 3.441   10  /opt/venv/lib/python3.9/site-packages/cleo/application.py:327 in run
#0 3.460        325│ 
#0 3.460        326│             try:
#0 3.460      → 327│                 exit_code = self._run(io)
#0 3.460        328│             except BrokenPipeError:
#0 3.460        329│                 # If we are piped to another process, it may close early and send a
#0 3.460 
#0 3.460    9  /opt/venv/lib/python3.9/site-packages/poetry/console/application.py:190 in _run
#0 3.473        188│         self._load_plugins(io)
#0 3.473        189│ 
#0 3.473      → 190│         exit_code: int = super()._run(io)
#0 3.473        191│         return exit_code
#0 3.473        192│ 
#0 3.473 
#0 3.473    8  /opt/venv/lib/python3.9/site-packages/cleo/application.py:431 in _run
#0 3.492        429│             io.input.interactive(interactive)
#0 3.492        430│ 
#0 3.492      → 431│         exit_code = self._run_command(command, io)
#0 3.492        432│         self._running_command = None
#0 3.492        433│ 
#0 3.492 
#0 3.492    7  /opt/venv/lib/python3.9/site-packages/cleo/application.py:473 in _run_command
#0 3.512        471│ 
#0 3.512        472│         if error is not None:
#0 3.512      → 473│             raise error
#0 3.512        474│ 
#0 3.512        475│         return terminate_event.exit_code
#0 3.512 
#0 3.512    6  /opt/venv/lib/python3.9/site-packages/cleo/application.py:457 in _run_command
#0 3.532        455│ 
#0 3.532        456│             if command_event.command_should_run():
#0 3.532      → 457│                 exit_code = command.run(io)
#0 3.532        458│             else:
#0 3.532        459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
#0 3.532 
#0 3.532    5  /opt/venv/lib/python3.9/site-packages/cleo/commands/base_command.py:119 in run
#0 3.537        117│         io.input.validate()
#0 3.537        118│ 
#0 3.537      → 119│         status_code = self.execute(io)
#0 3.537        120│ 
#0 3.537        121│         if status_code is None:
#0 3.537 
#0 3.537    4  /opt/venv/lib/python3.9/site-packages/cleo/commands/command.py:62 in execute
#0 3.546         60│ 
#0 3.546         61│         try:
#0 3.546      →  62│             return self.handle()
#0 3.546         63│         except KeyboardInterrupt:
#0 3.546         64│             return 1
#0 3.546 
#0 3.546    3  /opt/venv/lib/python3.9/site-packages/poetry/console/commands/install.py:187 in handle
#0 3.551        185│             return 0
#0 3.551        186│ 
#0 3.551      → 187│         builder.build()
#0 3.551        188│ 
#0 3.551        189│         if overwrite:
#0 3.551 
#0 3.551    2  /opt/venv/lib/python3.9/site-packages/poetry/masonry/builders/editable.py:63 in build
#0 3.560         61│                     "  - Falling back on using a setup.py"
#0 3.560         62│                 )
#0 3.560      →  63│                 self._setup_build()
#0 3.560         64│                 path: Path = self._path
#0 3.560         65│                 return path
#0 3.560 
#0 3.560    1  /opt/venv/lib/python3.9/site-packages/poetry/masonry/builders/editable.py:112 in _setup_build
#0 3.569        110│                 )
#0 3.569        111│                 try:
#0 3.569      → 112│                     pip_install(self._path, self._env, upgrade=True, editable=True)
#0 3.569        113│                 finally:
#0 3.569        114│                     shutil.move(
#0 3.569 
#0 3.569   PoetryException
#0 3.569 
#0 3.569   Failed to install /app
#0 3.569 
#0 3.569   at /opt/venv/lib/python3.9/site-packages/poetry/utils/pip.py:58 in pip_install
#0 3.570        54│ 
#0 3.571        55│     try:
#0 3.571        56│         return environment.run_pip(*args)
#0 3.571        57│     except EnvCommandError as e:
#0 3.571     →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
#0 3.571        59│ 
------
Dockerfile:26
--------------------
  24 |     WORKDIR /app
  25 |     RUN rm -f /app/id_rsa && poetry build 
  26 | >>> RUN poetry install --only-root -vvv
  27 |     
  28 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c poetry install --only-root -vvv" did not complete successfully: exit code: 1
make: *** [Makefile:21: build_image] Error 1
```

Resolving patch:
diff --git a/src/poetry/masonry/builders/editable.py b/src/poetry/masonry/builders/editable.py
--- a/src/poetry/masonry/builders/editable.py
+++ b/src/poetry/masonry/builders/editable.py
@@ -9,7 +9,6 @@
 from pathlib import Path
 from typing import TYPE_CHECKING
 
-from poetry.core.constraints.version import Version
 from poetry.core.masonry.builders.builder import Builder
 from poetry.core.masonry.builders.sdist import SdistBuilder
 from poetry.core.masonry.utils.package_include import PackageInclude
@@ -101,16 +100,7 @@ def _setup_build(self) -> None:
                 f.write(decode(builder.build_setup()))
 
         try:
-            if self._env.pip_version < Version.from_parts(19, 0):
-                pip_install(self._path, self._env, upgrade=True, editable=True)
-            else:
-                # Temporarily rename pyproject.toml
-                renamed_pyproject = self._poetry.file.path.with_suffix(".tmp")
-                self._poetry.file.path.rename(renamed_pyproject)
-                try:
-                    pip_install(self._path, self._env, upgrade=True, editable=True)
-                finally:
-                    renamed_pyproject.rename(self._poetry.file.path)
+            pip_install(self._path, self._env, upgrade=True, editable=True)
         finally:
             if not has_setup:
                 os.remove(setup)


Test patch: