Repository: jazzband/pip-tools

Issue:
pip-compile fails on packages using poetry
`pip-compile` fails if a package uses `poetry` which is PEP 517 compliant (https://poetry.eustace.io/docs/pyproject/#poetry-and-pep-517). 

Simply doing `pip install pendulum` does not require e.g. `poetry` to be installed. However, installing it made `pip-compile` work which I'm not sure we as consumers of the package in question should need to do.

##### Environment Versions

1. Alpine 3.8
1. Python version: `3.7.2`
1. pip version:  `19.0.1`
1. pip-tools version: `3.3.1`

##### Steps to replicate
```Dockerfile
# DOES NOT WORK

FROM python:3.7.2-alpine3.8

RUN pip3 install --upgrade pip==19.0.1 setuptools pip-tools>==3.3.1

RUN echo "pendulum" > req.txt

RUN pip-compile --generate-hashes req.txt


# WORKS

FROM python:3.7.2-alpine3.8

RUN pip3 install --upgrade pip==19.0.1 setuptools pip-tools>==3.3.1

RUN echo "pendulum" > req.txt

RUN pip3 install poetry

RUN pip-compile --generate-hashes req.txt
```


Discussion:
Hi @eliasbrange, thanks for the report.

I'm unfamiliar with PEP517 (/todo Read that thing!). I would naively wish for `pip` to handle the mechanic.

One thing to note, is that this specific install seem to work with `pip==18.1`.

Also, there's an issue on the subject in the pendulum project, about issues with `pip>=19.0.1`:
https://github.com/sdispater/pendulum/issues/335

Additionally, there seem to be some issues related to `pip 19.0.1` and `poetry` compatibility (I don't have quite the time to dig into these at the moment):
https://github.com/sdispater/poetry/issues

So there's the possibility that this is one of the upstreams issue, or that there's something more to do in `pip-tools` now to initialise some PEP517 mechanic (hunch from the `pip._vendor.pep517.wrappers.BackendUnavailable` exception raised, as seen when using `--verbose`)

Any help to untangle this would be greatly appreciated!
Trying to investigate a bit further. Noticed that `pip-compile` succeeds with `pytzdata` which also uses `poetry` as a build system: https://github.com/sdispater/pytzdata/blob/master/pyproject.toml
It raises here:
https://github.com/pypa/pip/blob/master/src/pip/_vendor/pep517/_in_process.py#L36

with an error:
> No module named 'poetry'

For some reason it wants `poetry`. Keep digging.
Which is described in `pendulum`s `pyproject.toml` as per PEP517/PEP518.

```
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
```
Right!

It doesn't go here:
https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/prepare.py#L119

because `should_isolate` is False (because build_isolation is False). So it cannot build requires. I've hardcoded `should_isolate=True` and `pip-compile` went fine.
The main problem is here https://github.com/jazzband/pip-tools/blob/5029fcc4298def004a3d286aa3adfc312acb4efd/piptools/repositories/pypi.py#L164
So that should probably be an optional flag to let consumers choose between installing required packages themselves, or doing isolated builds (which will take forever if there are lots of packages using different build systems).
let's investigate how `pip download pendulum` decides to build it in isolation mode. Ah it seems `pip download` builds in isolation mode by default.
> So that should probably be an optional flag to let consumers choose between installing required packages themselves, or doing isolated builds (which will take forever if there are lots of packages using different build systems).

I guess so. Build isolation could be set on by default and an option to disable it:
```
  --no-build-isolation        Disable isolation when building a modern source
```
Gentle ping to @vphilippon.

What do you think about this? Since pip builds sources in isolation by default, perhaps we should consider to do it as well. But it could be a "breaking change".
I went and read https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support to catch up a bit on the subject of build isolation and what it implies.

It seems `pip` enables build isolation since its first inclusion (https://github.com/pypa/pip/commit/c08d4cc806897de687785c723ac7b767c1bc2711)
So it seems reasonable to enable it by default in `pip-tools` too.

As for the "breaking change" nature of this change, I don't mind as it's meant to adapt to the "new reality and expectation" of the python packaging ecosystem.

The Grand Question: Do you think that:

A) It's actually a `pip-tools` bug we're only catching now, that having isolation disabled most likely caused more pain to users since `pip 10.0.0` release, and that enabling it will cause little surprise/impact to users? If so, include in our next feature/bugfix.

B) Enabling isolation by default may have multiple impacts that will require careful attention for a substantial part of the userbase, or could bring odd behavior in pip-tools itself, and thus should be announced strongly? If so, time for a 4.0.0 (which is ok, it's just numbers! Let's take advantage of Semantic Versionning to raise user awareness on this, as we can).

@atugushev @eliasbrange Voice your opinion, you likely have more info on the latest feature of pip than I do right now.
Also pingning @techalchemy and @pradyunsg for input on the impact/risk, if you have time 😄 .
A) I don't think it is a bug, "no isolation" might be chosen for a special reason. Would be appreciated to get a comment from @techalchemy here.

B) I totally agree. Users has to be warned if default behavior suddenly changes.

There is a soft way though. We could introduce `--no-build-isolation` and `--build-isolation` flags, where `--build-isolation` is default.
Seems related to pypa/pip#6164.

I'm inclined to think poetry needs to update its backend since it's not compatible with pip's PEP 517 implementation as things stand.
As for rollout of isolation to end users, @techalchemy can probably explain in better terms. :)
Just ran into this with a package of mine that uses `pyproject.toml` to declare some build system requirements.  Forcing build isolation as mentioned above fixed the issue.

Understand that a breaking change is problematic - is there a plan to at least add an optional flag to allow users to specify this behaviour if they need it?

Resolving patch:
diff --git a/piptools/repositories/pypi.py b/piptools/repositories/pypi.py
--- a/piptools/repositories/pypi.py
+++ b/piptools/repositories/pypi.py
@@ -49,9 +49,10 @@ class PyPIRepository(BaseRepository):
     config), but any other PyPI mirror can be used if index_urls is
     changed/configured on the Finder.
     """
-    def __init__(self, pip_options, session):
+    def __init__(self, pip_options, session, build_isolation=False):
         self.session = session
         self.pip_options = pip_options
+        self.build_isolation = build_isolation
 
         index_urls = [pip_options.index_url] + pip_options.extra_index_urls
         if pip_options.no_index:
@@ -161,7 +162,7 @@ def resolve_reqs(self, download_dir, ireq, wheel_cache):
                 'download_dir': download_dir,
                 'wheel_download_dir': self._wheel_download_dir,
                 'progress_bar': 'off',
-                'build_isolation': False
+                'build_isolation': self.build_isolation,
             }
             resolver_kwargs = {
                 'finder': self.finder,
diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -61,10 +61,14 @@
 @click.option('--max-rounds', default=10,
               help="Maximum number of rounds before resolving the requirements aborts.")
 @click.argument('src_files', nargs=-1, type=click.Path(exists=True, allow_dash=True))
+@click.option('--build-isolation/--no-build-isolation', is_flag=True, default=False,
+              help="Enable isolation when building a modern source distribution. "
+                   "Build dependencies specified by PEP 518 must be already installed "
+                   "if build isolation is disabled.")
 def cli(verbose, quiet, dry_run, pre, rebuild, find_links, index_url, extra_index_url,
         cert, client_cert, trusted_host, header, index, emit_trusted_host, annotate,
         upgrade, upgrade_packages, output_file, allow_unsafe, generate_hashes,
-        src_files, max_rounds):
+        src_files, max_rounds, build_isolation):
     """Compiles requirements.txt from requirements.in specs."""
     log.verbosity = verbose - quiet
 
@@ -122,7 +126,7 @@ def cli(verbose, quiet, dry_run, pre, rebuild, find_links, index_url, extra_inde
     pip_options, _ = pip_command.parse_args(pip_args)
 
     session = pip_command._build_session(pip_options)
-    repository = PyPIRepository(pip_options, session)
+    repository = PyPIRepository(pip_options, session, build_isolation)
 
     upgrade_install_reqs = {}
     # Proxy with a LocalRequirementsRepository if --upgrade is not specified


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -534,3 +534,21 @@ def test_cert_option(MockPyPIRepository, runner, option, attr, expected):
     for call in MockPyPIRepository.call_args_list:
         pip_options = call[0][0]
         assert getattr(pip_options, attr) == expected
+
+
+@pytest.mark.parametrize('option, expected', [
+    ('--build-isolation', True),
+    ('--no-build-isolation', False),
+])
+@mock.patch('piptools.scripts.compile.PyPIRepository')
+def test_build_isolation_option(MockPyPIRepository, runner, option, expected):
+    """
+    A value of the --build-isolation/--no-build-isolation flag must be passed to the PyPIRepository.
+    """
+    with open('requirements.in', 'w'):
+        pass
+
+    runner.invoke(cli, [option])
+
+    # Ensure the build_isolation option in PyPIRepository has the expected value.
+    assert [call[0][2] for call in MockPyPIRepository.call_args_list] == [expected]