Repository: jazzband/pip-tools

Issue:
via not shown for packages installed from VCS
my `requirements.in` file has:

```
-e git+https://github.com/scikit-image/scikit-image.git@33646e0#egg=scikit-image==0.12.git
```

but the `.txt` just shows:

```
dask[array]==0.7.6
```

without showing the `via`. This even though I gave the package a name with the `egg=`.

show via for direct deps
Currently when compiling, direct deps do not include a 'via' declaration. eg:

```
git grep "# via"
...
requirements.txt:    # via entrypoints, flake8, importlib-metadata, pylint
...
```

however it is missing from direct dependencies.

I propose:

```
# via ./requirements.in, etc, etc
```
for direct deps


Discussion:
FTR, I stumbled on the cause of this in the code: The editable dependencies are never added to the dependency cache (which is ok, editable packages requirements are likely to change for a same version), and the OutputWriter uses the reversed dependency cache to generate the "via" comments.
This probably isn't directly usable in pip-tools, but if it's any help, here's how I implemented this feature to Prequ: https://github.com/suutari/prequ/commit/634ef5c5daae3eb1911da2b482a766667d8d6364
@vphilippon I'd love to see this enhancement. If I submit a PR using the design from [above](https://github.com/suutari/prequ/commit/634ef5c5daae3eb1911da2b482a766667d8d6364), would that be welcome, i.e. filtering out editable dependencies at persistence time? Another idea is to maintain a separate in-memory-only cache for editables.
Or:

```
# directly and via etc, etc
```

to avoid squatting any possible future package name namespace
Hello @graingert! What do you mean by "direct deps"?
Deps discovered in the .in file
I see. Could you please provide a real example of the current behavior and what is you proposing? I want to be sure that i understand the issue.
The current behaviour is that some dependencies in the generated requirements.txt do not include a # via comment.  I'd prefer it if all dependencies included a # via comment
> The current behaviour is that some dependencies in the generated requirements.txt do not include a # via comment.

Could you show an example?
No it's not possible because the comment is missing
Okay, let's suppose:

```bash
$ cat requiremets.in
django

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
django==2.2.4
pytz==2019.2              # via django
sqlparse==0.3.0           # via django
```

The `django` is a "direct dependency" (or primary package). So, do you want to see "# via requirements.in" close to the "django" in requirements.txt? For example:

```bash
$ cat requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
django==2.2.4             # via requirements.in
pytz==2019.2              # via django
sqlparse==0.3.0           # via django
```
```
graingert@onomastic:~$ cat requirements.in 
django
django-debug-toolbar
graingert@onomastic:~$ pip-compile --generate-hashes requirements.in 
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes requirements.in
#
django-debug-toolbar==2.0 \
    --hash=sha256:17c53cd6bf4e7d69902aedf9a1d26c5d3b7369b54c5718744704f27b5a72f35d \
    --hash=sha256:9a23ada2e43cd989195db3c18710b5d7451134a0d48127ab64c1d2ad81700342
django==2.2.4 \
    --hash=sha256:16a5d54411599780ac9dfe3b9b38f90f785c51259a584e0b24b6f14a7f69aae8 \
    --hash=sha256:9a2f98211ab474c710fcdad29c82f30fc14ce9917c7a70c3682162a624de4035
pytz==2019.2 \
    --hash=sha256:26c0b32e437e54a18161324a2fca3c4b9846b74a8dccddd843113109e1116b32 \
    --hash=sha256:c894d57500a4cd2d5c71114aaab77dbab5eabd9022308ce5ac9bb93a60a6f0c7 \
    # via django
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
    # via django, django-debug-toolbar
```
Instead I'd like to see
```
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes requirements.in
#
django-debug-toolbar==2.0 \
    --hash=sha256:17c53cd6bf4e7d69902aedf9a1d26c5d3b7369b54c5718744704f27b5a72f35d \
    --hash=sha256:9a23ada2e43cd989195db3c18710b5d7451134a0d48127ab64c1d2ad81700342
django==2.2.4 \
    --hash=sha256:16a5d54411599780ac9dfe3b9b38f90f785c51259a584e0b24b6f14a7f69aae8 \
    --hash=sha256:9a2f98211ab474c710fcdad29c82f30fc14ce9917c7a70c3682162a624de4035
    # directly and via django-debug-toolbar
pytz==2019.2 \
    --hash=sha256:26c0b32e437e54a18161324a2fca3c4b9846b74a8dccddd843113109e1116b32 \
    --hash=sha256:c894d57500a4cd2d5c71114aaab77dbab5eabd9022308ce5ac9bb93a60a6f0c7 \
    # via django
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
    # via django, django-debug-toolbar
graingert@onomastic:~$ 
```
I would like to see a feature like this. It'd be nice to know the dependencies between my explicitly declared packages.

Resolving patch:
diff --git a/piptools/resolver.py b/piptools/resolver.py
--- a/piptools/resolver.py
+++ b/piptools/resolver.py
@@ -318,6 +318,8 @@ def get_best_match(self, ireq):
             )
         )
         best_match.comes_from = ireq.comes_from
+        if hasattr(ireq, "_source_ireqs"):
+            best_match._source_ireqs = ireq._source_ireqs
         return best_match
 
     def _iter_dependencies(self, ireq):
diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -327,10 +327,14 @@ def cli(
 
                 dist = run_setup(src_file)
                 tmpfile.write("\n".join(dist.install_requires))
+                comes_from = "{name} ({filename})".format(
+                    name=dist.get_name(), filename=src_file
+                )
             else:
                 tmpfile.write(sys.stdin.read())
+                comes_from = "-r -"
             tmpfile.flush()
-            constraints.extend(
+            reqs = list(
                 parse_requirements(
                     tmpfile.name,
                     finder=repository.finder,
@@ -338,6 +342,9 @@ def cli(
                     options=repository.options,
                 )
             )
+            for req in reqs:
+                req.comes_from = comes_from
+            constraints.extend(reqs)
         else:
             constraints.extend(
                 parse_requirements(
@@ -396,33 +403,6 @@ def cli(
     # Output
     ##
 
-    # Compute reverse dependency annotations statically, from the
-    # dependency cache that the resolver has populated by now.
-    #
-    # TODO (1a): reverse deps for any editable package are lost
-    #            what SHOULD happen is that they are cached in memory, just
-    #            not persisted to disk!
-    #
-    # TODO (1b): perhaps it's easiest if the dependency cache has an API
-    #            that could take InstallRequirements directly, like:
-    #
-    #              cache.set(ireq, ...)
-    #
-    #            then, when ireq is editable, it would store in
-    #
-    #              editables[egg_name][link_without_fragment] = deps
-    #              editables['pip-tools']['git+...ols.git@future'] = {
-    #                  'click>=3.0', 'six'
-    #              }
-    #
-    #            otherwise:
-    #
-    #              self[as_name_version_tuple(ireq)] = {'click>=3.0', 'six'}
-    #
-    reverse_dependencies = None
-    if annotate:
-        reverse_dependencies = resolver.reverse_dependencies(results)
-
     writer = OutputWriter(
         src_files,
         output_file,
@@ -444,8 +424,6 @@ def cli(
     writer.write(
         results=results,
         unsafe_requirements=resolver.unsafe_constraints,
-        reverse_dependencies=reverse_dependencies,
-        primary_packages=primary_packages,
         markers={
             key_from_ireq(ireq): ireq.markers for ireq in constraints if ireq.markers
         },
diff --git a/piptools/writer.py b/piptools/writer.py
--- a/piptools/writer.py
+++ b/piptools/writer.py
@@ -3,6 +3,8 @@
 import os
 from itertools import chain
 
+import six
+
 from .click import unstyle
 from .logging import log
 from .utils import (
@@ -36,6 +38,12 @@
 )
 
 
+def _comes_from_as_string(ireq):
+    if isinstance(ireq.comes_from, six.string_types):
+        return ireq.comes_from
+    return key_from_ireq(ireq.comes_from)
+
+
 class OutputWriter(object):
     def __init__(
         self,
@@ -125,19 +133,9 @@ def write_flags(self):
         if emitted:
             yield ""
 
-    def _iter_lines(
-        self,
-        results,
-        unsafe_requirements=None,
-        reverse_dependencies=None,
-        primary_packages=None,
-        markers=None,
-        hashes=None,
-    ):
+    def _iter_lines(self, results, unsafe_requirements=None, markers=None, hashes=None):
         # default values
         unsafe_requirements = unsafe_requirements or []
-        reverse_dependencies = reverse_dependencies or {}
-        primary_packages = primary_packages or []
         markers = markers or {}
         hashes = hashes or {}
 
@@ -169,11 +167,7 @@ def _iter_lines(
                     yield MESSAGE_UNHASHED_PACKAGE
                     warn_uninstallable = True
                 line = self._format_requirement(
-                    ireq,
-                    reverse_dependencies,
-                    primary_packages,
-                    markers.get(key_from_ireq(ireq)),
-                    hashes=hashes,
+                    ireq, markers.get(key_from_ireq(ireq)), hashes=hashes
                 )
                 yield line
             yielded = True
@@ -194,11 +188,7 @@ def _iter_lines(
                     yield comment("# {}".format(ireq_key))
                 else:
                     line = self._format_requirement(
-                        ireq,
-                        reverse_dependencies,
-                        primary_packages,
-                        marker=markers.get(ireq_key),
-                        hashes=hashes,
+                        ireq, marker=markers.get(ireq_key), hashes=hashes
                     )
                     yield line
 
@@ -209,41 +199,32 @@ def _iter_lines(
         if warn_uninstallable:
             log.warning(MESSAGE_UNINSTALLABLE)
 
-    def write(
-        self,
-        results,
-        unsafe_requirements,
-        reverse_dependencies,
-        primary_packages,
-        markers,
-        hashes,
-    ):
+    def write(self, results, unsafe_requirements, markers, hashes):
 
-        for line in self._iter_lines(
-            results,
-            unsafe_requirements,
-            reverse_dependencies,
-            primary_packages,
-            markers,
-            hashes,
-        ):
+        for line in self._iter_lines(results, unsafe_requirements, markers, hashes):
             log.info(line)
             if not self.dry_run:
                 self.dst_file.write(unstyle(line).encode("utf-8"))
                 self.dst_file.write(os.linesep.encode("utf-8"))
 
-    def _format_requirement(
-        self, ireq, reverse_dependencies, primary_packages, marker=None, hashes=None
-    ):
+    def _format_requirement(self, ireq, marker=None, hashes=None):
         ireq_hashes = (hashes if hashes is not None else {}).get(ireq)
 
         line = format_requirement(ireq, marker=marker, hashes=ireq_hashes)
 
-        if not self.annotate or key_from_ireq(ireq) in primary_packages:
+        if not self.annotate:
             return line
 
-        # Annotate what packages this package is required by
-        required_by = reverse_dependencies.get(ireq.name.lower(), [])
+        # Annotate what packages or reqs-ins this package is required by
+        required_by = set()
+        if hasattr(ireq, "_source_ireqs"):
+            required_by |= {
+                _comes_from_as_string(src_ireq)
+                for src_ireq in ireq._source_ireqs
+                if src_ireq.comes_from
+            }
+        elif ireq.comes_from:
+            required_by.add(_comes_from_as_string(ireq))
         if required_by:
             annotation = ", ".join(sorted(required_by))
             line = "{:24}{}{}".format(


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -41,20 +41,27 @@ def test_command_line_overrides_pip_conf(pip_with_index_conf, runner):
 
 
 def test_command_line_setuptools_read(pip_conf, runner):
-    package = open("setup.py", "w")
-    package.write(
-        dedent(
-            """\
-            from setuptools import setup
-            setup(install_requires=[])
-            """
+    with open("setup.py", "w") as package:
+        package.write(
+            dedent(
+                """\
+                from setuptools import setup
+                setup(
+                    name="fake-setuptools-a",
+                    install_requires=["small-fake-a==0.1"]
+                )
+                """
+            )
         )
-    )
-    package.close()
     out = runner.invoke(cli)
 
-    # check that pip-compile generated a configuration
     assert "This file is autogenerated by pip-compile" in out.stderr
+    assert (
+        "small-fake-a==0.1         # via fake-setuptools-a (setup.py)"
+        in out.stderr.splitlines()
+    )
+
+    # check that pip-compile generated a configuration file
     assert os.path.exists("requirements.txt")
 
 
@@ -363,11 +370,11 @@ def test_upgrade_packages_option(pip_conf, runner):
     with open("requirements.txt", "w") as req_in:
         req_in.write("small-fake-a==0.1\nsmall-fake-b==0.1")
 
-    out = runner.invoke(cli, ["-P", "small-fake-b"])
+    out = runner.invoke(cli, ["--no-annotate", "-P", "small-fake-b"])
 
     assert out.exit_code == 0
-    assert "small-fake-a==0.1" in out.stderr
-    assert "small-fake-b==0.3" in out.stderr
+    assert "small-fake-a==0.1" in out.stderr.splitlines()
+    assert "small-fake-b==0.3" in out.stderr.splitlines()
 
 
 def test_upgrade_packages_option_irrelevant(pip_conf, runner):
@@ -379,11 +386,11 @@ def test_upgrade_packages_option_irrelevant(pip_conf, runner):
     with open("requirements.txt", "w") as req_in:
         req_in.write("small-fake-a==0.1")
 
-    out = runner.invoke(cli, ["--upgrade-package", "small-fake-b"])
+    out = runner.invoke(cli, ["--no-annotate", "--upgrade-package", "small-fake-b"])
 
     assert out.exit_code == 0
     assert "small-fake-a==0.1" in out.stderr.splitlines()
-    assert "small-fake-b==0.3" not in out.stderr
+    assert "small-fake-b==0.3" not in out.stderr.splitlines()
 
 
 def test_upgrade_packages_option_no_existing_file(pip_conf, runner):
@@ -394,11 +401,11 @@ def test_upgrade_packages_option_no_existing_file(pip_conf, runner):
     with open("requirements.in", "w") as req_in:
         req_in.write("small-fake-a\nsmall-fake-b")
 
-    out = runner.invoke(cli, ["-P", "small-fake-b"])
+    out = runner.invoke(cli, ["--no-annotate", "-P", "small-fake-b"])
 
     assert out.exit_code == 0
-    assert "small-fake-a==0.2" in out.stderr
-    assert "small-fake-b==0.3" in out.stderr
+    assert "small-fake-a==0.2" in out.stderr.splitlines()
+    assert "small-fake-b==0.3" in out.stderr.splitlines()
 
 
 @pytest.mark.parametrize(
@@ -502,7 +509,7 @@ def test_generate_hashes_with_editable(pip_conf, runner):
     small_fake_package_url = path_to_url(small_fake_package_dir)
     with open("requirements.in", "w") as fp:
         fp.write("-e {}\n".format(small_fake_package_url))
-    out = runner.invoke(cli, ["--generate-hashes"])
+    out = runner.invoke(cli, ["--no-annotate", "--generate-hashes"])
     expected = (
         "-e {}\n"
         "small-fake-a==0.1 \\\n"
@@ -523,7 +530,7 @@ def test_generate_hashes_with_url(runner):
             "https://github.com/jazzband/pip-tools/archive/"
             "7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools\n"
         )
-    out = runner.invoke(cli, ["--generate-hashes"])
+    out = runner.invoke(cli, ["--no-annotate", "--generate-hashes"])
     expected = (
         "https://github.com/jazzband/pip-tools/archive/"
         "7d86c8d3ecd1faa6be11c7ddc6b29a30ffd1dae3.zip#egg=pip-tools \\\n"
@@ -628,7 +635,7 @@ def test_stdin(pip_conf, runner):
         cli, ["-", "--output-file", "requirements.txt", "-n"], input="small-fake-a==0.1"
     )
 
-    assert "small-fake-a==0.1" in out.stderr
+    assert "small-fake-a==0.1         # via -r -" in out.stderr.splitlines()
 
 
 def test_multiple_input_files_without_output_file(runner):
@@ -652,7 +659,11 @@ def test_multiple_input_files_without_output_file(runner):
 @pytest.mark.parametrize(
     "option, expected",
     [
-        ("--annotate", "small-fake-a==0.1         # via small-fake-with-deps\n"),
+        (
+            "--annotate",
+            "small-fake-a==0.1         "
+            "# via -c constraints.txt (line 1), small-fake-with-deps\n",
+        ),
         ("--no-annotate", "small-fake-a==0.1\n"),
     ],
 )
@@ -660,7 +671,10 @@ def test_annotate_option(pip_conf, runner, option, expected):
     """
     The output lines has have annotations if option is turned on.
     """
+    with open("constraints.txt", "w") as constraints_in:
+        constraints_in.write("small-fake-a==0.1")
     with open("requirements.in", "w") as req_in:
+        req_in.write("-c constraints.txt\n")
         req_in.write("small_fake_with_deps")
 
     out = runner.invoke(cli, [option, "-n"])
@@ -681,7 +695,7 @@ def test_allow_unsafe_option(pip_conf, monkeypatch, runner, option, expected):
     with open("requirements.in", "w") as req_in:
         req_in.write(path_to_url(os.path.join(PACKAGES_PATH, "small_fake_with_deps")))
 
-    out = runner.invoke(cli, [option] if option else [])
+    out = runner.invoke(cli, ["--no-annotate", option] if option else [])
 
     assert expected in out.stderr.splitlines()
     assert out.exit_code == 0
@@ -746,7 +760,7 @@ def test_pre_option(pip_conf, runner, cli_option, infile_option, expected_packag
             req_in.write("--pre\n")
         req_in.write("small-fake-a\n")
 
-    out = runner.invoke(cli, ["-n"] + (["-p"] if cli_option else []))
+    out = runner.invoke(cli, ["--no-annotate", "-n"] + (["-p"] if cli_option else []))
 
     assert out.exit_code == 0, out.stderr
     assert expected_package in out.stderr.splitlines(), out.stderr
@@ -770,7 +784,7 @@ def test_dry_run_option(pip_conf, runner, add_options):
     with open("requirements.in", "w") as req_in:
         req_in.write("small-fake-a\n")
 
-    out = runner.invoke(cli, ["--dry-run"] + add_options)
+    out = runner.invoke(cli, ["--no-annotate", "--dry-run"] + add_options)
 
     assert out.exit_code == 0, out.stderr
     assert "small-fake-a==0.2" in out.stderr.splitlines()
@@ -805,7 +819,7 @@ def test_dry_run_doesnt_touch_output_file(
 
     before_compile_mtime = os.stat("requirements.txt").st_mtime
 
-    out = runner.invoke(cli, ["--dry-run"] + add_options)
+    out = runner.invoke(cli, ["--no-annotate", "--dry-run"] + add_options)
 
     assert out.exit_code == 0, out.stderr
     assert expected_cli_output_package in out.stderr.splitlines()
@@ -863,7 +877,6 @@ def test_upgrade_package_doesnt_remove_annotation(pip_conf, runner):
         )
 
     runner.invoke(cli, ["-P", "small-fake-a"])
-
     with open("requirements.txt", "r") as req_txt:
         assert (
             "small-fake-a==0.1         # via small-fake-with-deps"
@@ -1029,14 +1042,14 @@ def test_sub_dependencies_with_constraints(pip_conf, runner):
         req_in.write("-c constraints.txt\n")
         req_in.write("small_fake_with_deps_and_sub_deps")  # require fake package
 
-    out = runner.invoke(cli)
+    out = runner.invoke(cli, ["--no-annotate"])
 
     assert out.exit_code == 0
 
     req_out_lines = set(out.stderr.splitlines())
     assert {
-        "small-fake-a==0.1         # via small-fake-with-unpinned-deps",
-        "small-fake-b==0.2         # via small-fake-with-unpinned-deps",
+        "small-fake-a==0.1",
+        "small-fake-b==0.2",
         "small-fake-with-deps-and-sub-deps==0.1",
-        "small-fake-with-unpinned-deps==0.1  # via small-fake-with-deps-and-sub-deps",
+        "small-fake-with-unpinned-deps==0.1",
     }.issubset(req_out_lines)
diff --git a/tests/test_writer.py b/tests/test_writer.py
--- a/tests/test_writer.py
+++ b/tests/test_writer.py
@@ -49,54 +49,48 @@ def writer(tmpdir_cwd):
 def test_format_requirement_annotation_editable(from_editable, writer):
     # Annotations are printed as comments at a fixed column
     ireq = from_editable("git+git://fake.org/x/y.git#egg=y")
-    reverse_dependencies = {"y": ["xyz"]}
+    ireq.comes_from = "xyz"
 
     assert writer._format_requirement(
-        ireq, reverse_dependencies, primary_packages=[]
+        ireq
     ) == "-e git+git://fake.org/x/y.git#egg=y  " + comment("# via xyz")
 
 
 def test_format_requirement_annotation(from_line, writer):
     ireq = from_line("test==1.2")
-    reverse_dependencies = {"test": ["xyz"]}
+    ireq.comes_from = "xyz"
 
-    assert writer._format_requirement(
-        ireq, reverse_dependencies, primary_packages=[]
-    ) == "test==1.2                 " + comment("# via xyz")
+    assert writer._format_requirement(ireq) == "test==1.2                 " + comment(
+        "# via xyz"
+    )
 
 
 def test_format_requirement_annotation_lower_case(from_line, writer):
     ireq = from_line("Test==1.2")
-    reverse_dependencies = {"test": ["xyz"]}
+    ireq.comes_from = "xyz"
 
-    assert writer._format_requirement(
-        ireq, reverse_dependencies, primary_packages=[]
-    ) == "test==1.2                 " + comment("# via xyz")
+    assert writer._format_requirement(ireq) == "test==1.2                 " + comment(
+        "# via xyz"
+    )
 
 
-def test_format_requirement_not_for_primary(from_line, writer):
-    "Primary packages should not get annotated."
+def test_format_requirement_for_primary(from_line, writer):
+    "Primary packages should get annotated."
     ireq = from_line("test==1.2")
-    reverse_dependencies = {"test": ["xyz"]}
+    ireq.comes_from = "xyz"
 
-    assert (
-        writer._format_requirement(
-            ireq, reverse_dependencies, primary_packages=["test"]
-        )
-        == "test==1.2"
+    assert writer._format_requirement(ireq) == "test==1.2                 " + comment(
+        "# via xyz"
     )
 
 
-def test_format_requirement_not_for_primary_lower_case(from_line, writer):
-    "Primary packages should not get annotated."
+def test_format_requirement_for_primary_lower_case(from_line, writer):
+    "Primary packages should get annotated."
     ireq = from_line("Test==1.2")
-    reverse_dependencies = {"test": ["xyz"]}
+    ireq.comes_from = "xyz"
 
-    assert (
-        writer._format_requirement(
-            ireq, reverse_dependencies, primary_packages=["test"]
-        )
-        == "test==1.2"
+    assert writer._format_requirement(ireq) == "test==1.2                 " + comment(
+        "# via xyz"
     )
 
 
@@ -105,11 +99,8 @@ def test_format_requirement_environment_marker(from_line, writer):
     ireq = from_line(
         'test ; python_version == "2.7" and platform_python_implementation == "CPython"'
     )
-    reverse_dependencies = set()
 
-    result = writer._format_requirement(
-        ireq, reverse_dependencies, primary_packages=["test"], marker=ireq.markers
-    )
+    result = writer._format_requirement(ireq, marker=ireq.markers)
     assert (
         result == 'test ; python_version == "2.7" and '
         'platform_python_implementation == "CPython"'