Repository: jazzband/pip-tools

Issue:
Refactor parsing setup.py in favour of pip install -e .
#### What's the problem this feature will solve?
<!-- What are you trying to do, that you are unable to achieve with pip-tools as it currently stands? -->

The current implementation of `pip-compile setup.py` doesn't handle the following use-cases and errors:

1) doesn't support extra requirements, see #625
2) doesn't support environment markers, see #1139
5) doesn't support declarative config `setup.cfg`
6) doesn't support modern `pyproject.toml` config
3) suppresses an error message when setup.py parse failed, see #572 #684
4) raises an unexpected error during parsing a valid setup.py, see #511 #695

The solution based on `-e .` became "de facto" workaround, so why can't we handle it by pip-compile itself?

#### Describe the solution you'd like
<!-- A clear and concise description of what you want to happen. -->

Instead of:
```python
from distutils.core import run_setup
dist = run_setup(src_file)
tmpfile.write("\n".join(dist.install_requires))
```

get dependencies using something like (consider bash onliner as a demonstration):

```bash
$ echo "-e ." | pip-compile - -qo- | sed '/^-e / d' > tmpfile
```

<!-- Provide examples of real-world use cases that this would enable and how it solves the problem described above. -->

For example:
```bash
$ cat setup.py
from setuptools import setup
setup(
    name="mypackage",
    version="0.1",
    install_requires=['django']
)

$ echo "-e ." | pip-compile - -qo- | sed '/^-e / d'
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file=- -
#
django==2.2.5
pytz==2019.2              # via django
sqlparse==0.3.0           # via django
```

#### Alternative Solutions
<!-- Have you tried to work around the problem using pip-tools or other tools? Or a different approach to solving this issue? Please elaborate here. -->

Use [pip's internal api](https://github.com/pypa/pip/blob/b96ee36ba9c5d7e79592eab52c653ae9a37dbcb8/src/pip/_internal/legacy_resolve.py#L364-L367) to get dependencies from `setup.py`:

```python
...
abstract_dist = pip._internal.legacy_resolver.Resolver._get_abstract_dist_for(req_to_install)

# Parse and return dependencies
dist = abstract_dist.get_pkg_resources_distribution()
...
```

But i found it challenging to maintain, especially for old pip versions.

#### Additional context
<!-- Add any other context, links, etc. about the feature here. -->
N/A


**UPDATE 2020-05-16**: The downside is that `-e .` solution loses the environment markers.



Discussion:
I believe there is another issue where it currently doesn't handle `extras_require`. Could this solution be extended to handle that as well? I think it could, for example instead of running `pip-compile` with `-e .`, it'd run it with (for example), `-e .[extra1,extra2]`.
@barrywhart 

I think a new cli argument could do it. For example, `pip-compile setup.py --extras=extra1,extra2` or `pip-compile setup.py --extra=extra1 --extra=extra2`.

That would definitely be helpful.  I'm currently bumping up against `pip-sync` removing all of my packages installed via `pip install -e .[dev]`
UPDATE: added more use-cases to the issue:

> 4) doesn't support declarative config `setup.cfg`
> 5) doesn't support modern `pyproject.toml` config
Since we drop the support of old pip versions (see #926) I think better to go for the alternative solution and use pip internals.
EDIT: I wrote this whole thing, skipping this important part of the initial issue (sorry!):

> The solution based on -e . became "de facto" workaround, so why can't we handle it by pip-compile itself?

I basically agree on the idea behind that, absolutely.
One question though: why attempt to ommit the `-e <path>` from the resulting file? The following `pip install` should still be told to process it (say, for build dependencies, or actually performing some build step), no?

At this point, I'd suggest to work on making piping (`|`) `pip-compile` output directly to `pip install` or `pip-sync` more user friendly, or something along those line to skip the `requirements.in` containing just `-e .`.

Example:
`echo "-e ." | pip-compile - -qo- | pip install`
or something along those line? It's not so much about keeping a dependency list, rather that benefiting from the dependency resolver until `pip` has its own.

Original wall-of-text-below, still might be relevant info, will leave it there. Consider it an argument against directly processing `setup.py`/`setup.cfg`/`pyproject.toml` as input.

---

I'll share what might be an unpopular and purist opinion: I really don't like the fact that we support `setup.py` as direct input. Over time, I've come to consider it a misfeature.

My purist argument boils down to this article on `setup.py` vs `requirements.txt`, or Abstract vs Concrete dependencies, by Donald Stufft (I've brought that up in multiple discussion before, and I will again :) ):
https://caremad.io/posts/2013/07/setup-vs-requirement/

In short: The distribution package is not the app.

You don't need a `setup.py` file if you're not making a distribution package.
And I believe the same argument applies to `setup.cfg` and `pyproject.toml` files.
If this statement sounds wrong to you, please quote and give counter examples! I'd love to learn more.

---

For the case of the distribution package dev setup, to use `pip-tools` to manage it you can still think of it as "defining an app using your package".

dev-requirements.in:
```
-e .[dev]  # See how you don't have any issues specifying your extra?
```
and now your devs/contributors can run `pip-compile dev-requirements.in` and `pip install -r dev-requirements.txt` (or `pip-sync dev-requirements.txt`).
There's no redundancy here, and you've used a requirements file to define the concrete dependencies (which is its role) for an install.

The point, once again, is to distinguish the distribution package from the thing you want to install, which is a list of concrete dependencies, which includes your package.

When installing your package for dev, you don't run `pip install -r setup.py`.
No, you run `pip install -e .`, and I believe that things should go the same way with `pip-tools`.
You can try to craft a one-liner for doing the equivalent with `pip-compile`, or we can try to improve `pip-tools` to facilitate piping (`|`) `pip-compile` output to `pip install` or `pip-sync`, but using a requirements file stills seems like the cleaner and less error prone technique so far.

---

TL;DR This feature seems hard to get right (ex: support extras) because it tries to do the wrong job IMO. `pip install -r setup.py` isn't a thing, and for a good reason.

Should we remove that feature: I'd like to, but we might be doing more wrong than right.
Should we put energy in maintaining, improving and promoting this feature and usage: IMO, No.
@vphilippon 

You've [previously](https://github.com/jazzband/pip-tools/issues/625#issuecomment-376209060) made your opinions / dislike of setup.py known.

As I also [previously mentioned](https://github.com/jazzband/pip-tools/issues/625#issuecomment-544707338), adding the package itself to `requirements.in` doesn't make sense.

There is considerable value to using setup.py to establish the baseline requirements, along with the requirements file being used for freezing the requirements according to that specific machine / ci workflow.

I feel like the people who don't see the value to setup.py aren't actively making packages that use it.
@eedwards-sk 

Thanks for the input, and bringing this previous mention:
>As I also previously mentioned, adding the package itself to requirements.in doesn't make sense.

I'm still not sure I see how adding the package itself to `requirements.in` doesn't make sense.
To the contrary (as I mentioned in my later edit above, sorry about that), the generated `requirements.txt` should (must?) include a reference the the package itself, no? Otherwise, running `pip install -r requirements.txt` actually won't process the build steps required by the package itself to be runable afterwards (ex: c++ compilation). 
See my last [comment](https://github.com/jazzband/pip-tools/issues/625#issuecomment-545036891) in that thread

It's a problem. Not sure why you haven't ran into this yet -- but I don't want my package getting added as a dependency of itself. It makes no sense!
@eedwards-sk 

> Additionally, -e . does not work with --generate-hashes .

I do not usually use `--generate-hashes`, and after testing, I see how using `-e .` and `--require-hashes` is an issue, you're right.
So unless there's a way for `pip-tools` to have a path to the package that is hashable (is it even possible at all?), there is an absolute no-go there. TIL.

> There is considerable value to using setup.py to establish the baseline requirements, along with the requirements file being used for freezing the requirements according to that specific machine / ci workflow.

>Yep. I have a package that should be able to be 'frozen' with hashes, for both its regular dependencies, and foo[dev]. Adding the package itself as a dependency into requirements.txt makes no sense.

Thinking about it over again, I guess I can see some example such as using `extras_require` to define dev requirements. The fact that there are cases for packages with a `setup.py` that we actively don't want to process with `pip install` kind of baffles me, but it looks like it's there.

@atugushev's idea of handling this via an equivalent to `-e .` internally and remove the package itself make the most sense I guess.
In that case, I'd find it nice to be able to do something like `pip-compile .[dev] -o dev-requirements.txt` rather than adding a parameter for extras requires, but I feel like this might become troublesome.
Thanks for taking the time to learn more about the issue.
@vphilippon thanks for sharing your thoughts!

> In that case, I'd find it nice to be able to do something like `pip-compile .[dev] -o dev-requirements.txt` rather than adding a parameter for extras requires, but I feel like this might become troublesome.

The idea of `pip-compile .[dev] -o dev-requirements.txt` sounds good to me. This API looks better than `pip-install [setup.py|setup.cfg|pyproject.toml]` (see  #1047 for `setup.cfg` and `pyproject.toml`). I would even use an `-e` flag for editables to be more consistent with `pip`, e.g. `pip-compile -e .[dev] -o dev-requirements.txt`.
it may be better to use "pep517.meta.load" from `pip install pep517`
One comment from the peanut gallery:

> I think better to go for the alternative solution and use pip internals.

pip developers are generally not happy with others doing this. pip does not have an API, the devs have repeatedly stated so in several places [citation needed] and regardless, any solution regarding on pip internals will be very fragile.
@astrojuanlu pip-tools makes heavy use of pip internals already
The downside of using `-e .` is that it loses the environment markers. I've updated the issue.
> it may be better to use "pep517.meta.load" from pip install pep517

@jaraco agrees with @graingert 

https://github.com/pypa/setuptools/issues/1951#issuecomment-722645440

And it seems to retain environment markers.

Resolving patch:
diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -7,8 +7,12 @@
 import click
 from click.utils import safecall
 from pip._internal.commands import create_command
-from pip._internal.req.constructors import install_req_from_line
+from pip._internal.req.constructors import (
+    install_req_from_line,
+    install_req_from_req_string,
+)
 from pip._internal.utils.misc import redact_auth_from_url
+from pip._vendor.pep517 import meta
 
 from .._compat import parse_requirements
 from ..cache import DependencyCache
@@ -330,21 +334,14 @@ def cli(
     constraints = []
     for src_file in src_files:
         is_setup_file = os.path.basename(src_file) == "setup.py"
-        if is_setup_file or src_file == "-":
+        if src_file == "-":
             # pip requires filenames and not files. Since we want to support
             # piping from stdin, we need to briefly save the input from stdin
             # to a temporary file and have pip read that.  also used for
             # reading requirements from install_requires in setup.py.
             tmpfile = tempfile.NamedTemporaryFile(mode="wt", delete=False)
-            if is_setup_file:
-                from distutils.core import run_setup
-
-                dist = run_setup(src_file)
-                tmpfile.write("\n".join(dist.install_requires))
-                comes_from = f"{dist.get_name()} ({src_file})"
-            else:
-                tmpfile.write(sys.stdin.read())
-                comes_from = "-r -"
+            tmpfile.write(sys.stdin.read())
+            comes_from = "-r -"
             tmpfile.flush()
             reqs = list(
                 parse_requirements(
@@ -357,6 +354,15 @@ def cli(
             for req in reqs:
                 req.comes_from = comes_from
             constraints.extend(reqs)
+        elif is_setup_file:
+            dist = meta.load(os.path.dirname(os.path.abspath(src_file)))
+            comes_from = f"{dist.metadata.get_all('Name')[0]} ({src_file})"
+            constraints.extend(
+                [
+                    install_req_from_req_string(req, comes_from=comes_from)
+                    for req in dist.requires or []
+                ]
+            )
         else:
             constraints.extend(
                 parse_requirements(
@@ -378,7 +384,13 @@ def cli(
 
     # Filter out pip environment markers which do not match (PEP496)
     constraints = [
-        req for req in constraints if req.markers is None or req.markers.evaluate()
+        req
+        for req in constraints
+        if req.markers is None
+        # We explicitly set extra=None to filter out optional requirements
+        # since evaluating an extra marker with no environment raises UndefinedEnvironmentName
+        # (see https://packaging.pypa.io/en/latest/markers.html#usage)
+        or req.markers.evaluate({"extra": None})
     ]
 
     log.debug("Using indexes:")


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -38,7 +38,17 @@ def test_command_line_overrides_pip_conf(pip_with_index_conf, runner):
     assert "Using indexes:\n  http://override.com" in out.stderr
 
 
-def test_command_line_setuptools_read(pip_conf, runner):
+@pytest.mark.network
+def test_command_line_setuptools_read(runner, make_pip_conf):
+    make_pip_conf(
+        dedent(
+            """\
+            [global]
+            disable-pip-version-check = True
+            """
+        )
+    )
+
     with open("setup.py", "w") as package:
         package.write(
             dedent(
@@ -51,16 +61,13 @@ def test_command_line_setuptools_read(pip_conf, runner):
                 """
             )
         )
-    out = runner.invoke(cli, ["--no-emit-find-links"])
+    out = runner.invoke(
+        cli,
+        ["--no-header", "--no-emit-find-links", "--find-links", MINIMAL_WHEELS_PATH],
+    )
 
     assert out.stderr == dedent(
         """\
-        #
-        # This file is autogenerated by pip-compile
-        # To update, run:
-        #
-        #    pip-compile --no-emit-find-links
-        #
         small-fake-a==0.1
             # via fake-setuptools-a (setup.py)
         """
@@ -70,6 +77,7 @@ def test_command_line_setuptools_read(pip_conf, runner):
     assert os.path.exists("requirements.txt")
 
 
+@pytest.mark.network
 @pytest.mark.parametrize(
     ("options", "expected_output_file"),
     (
@@ -85,12 +93,11 @@ def test_command_line_setuptools_read(pip_conf, runner):
         (["setup.py", "--output-file", "output.txt"], "output.txt"),
     ),
 )
-def test_command_line_setuptools_output_file(
-    pip_conf, runner, options, expected_output_file
-):
+def test_command_line_setuptools_output_file(runner, options, expected_output_file):
     """
     Test the output files for setup.py as a requirement file.
     """
+
     with open("setup.py", "w") as package:
         package.write(
             dedent(
@@ -106,7 +113,8 @@ def test_command_line_setuptools_output_file(
     assert os.path.exists(expected_output_file)
 
 
-def test_command_line_setuptools_nested_output_file(pip_conf, tmpdir, runner):
+@pytest.mark.network
+def test_command_line_setuptools_nested_output_file(tmpdir, runner):
     """
     Test the output file for setup.py in nested folder as a requirement file.
     """
@@ -127,6 +135,43 @@ def test_command_line_setuptools_nested_output_file(pip_conf, tmpdir, runner):
     assert (proj_dir / "requirements.txt").exists()
 
 
+@pytest.mark.network
+def test_setuptools_preserves_environment_markers(
+    runner, make_package, make_wheel, make_pip_conf, tmpdir
+):
+    make_pip_conf(
+        dedent(
+            """\
+            [global]
+            disable-pip-version-check = True
+            """
+        )
+    )
+
+    dists_dir = tmpdir / "dists"
+
+    foo_dir = make_package(name="foo", version="1.0")
+    make_wheel(foo_dir, dists_dir)
+
+    bar_dir = make_package(
+        name="bar", version="2.0", install_requires=['foo ; python_version >= "1"']
+    )
+    out = runner.invoke(
+        cli,
+        [
+            str(bar_dir / "setup.py"),
+            "--no-header",
+            "--no-annotate",
+            "--no-emit-find-links",
+            "--find-links",
+            str(dists_dir),
+        ],
+    )
+
+    assert out.exit_code == 0, out.stderr
+    assert out.stderr == 'foo==1.0 ; python_version >= "1"\n'
+
+
 def test_find_links_option(runner):
     with open("requirements.in", "w") as req_in:
         req_in.write("-f ./libs3")