Repository: jazzband/pip-tools

Issue:
5.1.2 regression: lost dependencies using same zip file with different subdirectories
In `pip-tools` 5.1.2 and later, when I provide two copies of the same zip file URL with different subdirectories, the dependencies of the first are lost:

```
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip==0.7.0&subdirectory=zulip
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip-bots==0.7.0&subdirectory=zulip_bots
```

`git bisect` shows that this regression was introduced in commit c0b33e7f2224758524001e755cc338965ca0b2c0 (#1132).

Cc @richafrank

#### Environment Versions

1. OS Type: Ubuntu 18.04
1. Python version: 3.6.9
1. pip version: 20.1.1
1. pip-tools version: 5.2.0

#### Steps to replicate

```sh
cat > z.in <<EOF
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip==0.7.0&subdirectory=zulip
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip-bots==0.7.0&subdirectory=zulip_bots
EOF
pip-compile z.in
```

#### Expected result

```
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile z.in
#
beautifulsoup4==4.9.1     # via zulip-bots
certifi==2020.4.5.1       # via requests
cffi==1.14.0              # via cryptography
chardet==3.0.4            # via requests
cryptography==2.9.2       # via pyopenssl, requests
distro==1.5.0             # via zulip
html2text==2020.1.16      # via zulip-bots
idna==2.9                 # via requests
lxml==4.5.1               # via zulip-bots
matrix-client==0.3.2      # via zulip
pycparser==2.20           # via cffi
pyopenssl==19.1.0         # via requests
requests[security]==2.23.0  # via matrix-client, zulip
six==1.15.0               # via cryptography, pyopenssl, zulip
soupsieve==2.0.1          # via beautifulsoup4
urllib3==1.25.9           # via requests
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip-bots==0.7.0&subdirectory=zulip_bots  # via -r z.in
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip==0.7.0&subdirectory=zulip  # via -r z.in, zulip-bots

# The following packages are considered to be unsafe in a requirements file:
# pip
```

#### Actual result

```
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile z.in
#
beautifulsoup4==4.9.1     # via zulip-bots
html2text==2020.1.16      # via zulip-bots
lxml==4.5.1               # via zulip-bots
soupsieve==2.0.1          # via beautifulsoup4
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip-bots==0.7.0&subdirectory=zulip_bots  # via -r z.in
https://github.com/zulip/python-zulip-api/archive/0.7.0.zip#egg=zulip==0.7.0&subdirectory=zulip  # via -r z.in, zulip-bots

# The following packages are considered to be unsafe in a requirements file:
# pip
```



Discussion:
I haven’t figured out why, but it works correctly if I delete this optimization: 

```diff
--- a/piptools/resolver.py
+++ b/piptools/resolver.py
@@ -55,10 +55,6 @@ def combine_install_requirements(repository, ireqs):
     for ireq in ireqs:
         source_ireqs.extend(getattr(ireq, "_source_ireqs", [ireq]))
 
-    # Optimization. Don't bother with combination logic.
-    if len(source_ireqs) == 1:
-        return source_ireqs[0]
-
     # deepcopy the accumulator so as to not modify the inputs
     combined_ireq = copy.deepcopy(source_ireqs[0])
     repository.copy_ireq_dependencies(source_ireqs[0], combined_ireq)
```

but only with `egg=` in the URLs.
Related to #1154.
That the optimization breaks it is surprising, but this whole combination step keeps yielding many surprises... 

If no one else has figured it out, I'll take a look at this and the related issue as well, hopefully in the next week.
@richafrank thank you!

Resolving patch:
diff --git a/piptools/repositories/base.py b/piptools/repositories/base.py
--- a/piptools/repositories/base.py
+++ b/piptools/repositories/base.py
@@ -45,6 +45,7 @@ def allow_all_wheels(self):
         Monkey patches pip.Wheel to allow wheels from all platforms and Python versions.
         """
 
+    @abstractmethod
     def copy_ireq_dependencies(self, source, dest):
         """
         Notifies the repository that `dest` is a copy of `source`, and so it
diff --git a/piptools/repositories/local.py b/piptools/repositories/local.py
--- a/piptools/repositories/local.py
+++ b/piptools/repositories/local.py
@@ -92,3 +92,6 @@ def get_hashes(self, ireq):
     def allow_all_wheels(self):
         with self.repository.allow_all_wheels():
             yield
+
+    def copy_ireq_dependencies(self, source, dest):
+        self.repository.copy_ireq_dependencies(source, dest)


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -88,6 +88,10 @@ def allow_all_wheels(self):
         # No need to do an actual pip.Wheel mock here.
         yield
 
+    def copy_ireq_dependencies(self, source, dest):
+        # No state to update.
+        pass
+
 
 class FakeInstalledDistribution(object):
     def __init__(self, line, deps=None):
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -1308,3 +1308,28 @@ def test_prefer_binary_dist_even_there_is_source_dists(
 
     assert out.exit_code == 0, out
     assert "test-package==2.0" in out.stderr.splitlines(), out.stderr
+
+
+@pytest.mark.parametrize("output_content", ("small_fake_with_deps", ""))
+def test_duplicate_reqs_combined(pip_conf, runner, output_content):
+    """
+    Test pip-compile tracks dependencies properly when install requirements are
+    combined, especially when an output file already exists.
+
+    Regression test for issue GH-1154.
+    """
+    fake_package_dir = os.path.join(PACKAGES_PATH, "small_fake_with_deps")
+    fake_package_dir = path_to_url(fake_package_dir)
+    with open("requirements.in", "w") as req_in:
+        req_in.write(fake_package_dir + "\n")
+        req_in.write(fake_package_dir + "#egg=small_fake_with_deps\n")
+
+    if output_content:
+        with open("requirements.txt", "w") as reqs_out:
+            reqs_out.write(output_content)
+
+    out = runner.invoke(cli, [])
+
+    assert out.exit_code == 0, out
+    assert fake_package_dir in out.stderr
+    assert "small-fake-a==0.1" in out.stderr
diff --git a/tests/test_repository_local.py b/tests/test_repository_local.py
--- a/tests/test_repository_local.py
+++ b/tests/test_repository_local.py
@@ -1,3 +1,7 @@
+import copy
+
+from tests.conftest import FakeRepository
+
 from piptools.repositories.local import LocalRequirementsRepository
 from piptools.utils import name_from_req
 
@@ -23,3 +27,25 @@ def test_get_hashes_local_repository_cache_hit(from_line, repository):
     with local_repository.allow_all_wheels():
         hashes = local_repository.get_hashes(from_line("small-fake-a==0.1"))
         assert hashes == EXPECTED
+
+
+class FakeRepositoryChecksForCopy(FakeRepository):
+    def __init__(self):
+        super(FakeRepositoryChecksForCopy, self).__init__()
+        self.copied = []
+
+    def copy_ireq_dependencies(self, source, dest):
+        self.copied.append(source)
+
+
+def test_local_repository_copy_ireq_dependencies(from_line):
+    # Ensure that local repository forwards any messages to update its state
+    # of ireq dependencies.
+    checker = FakeRepositoryChecksForCopy()
+    local_repository = LocalRequirementsRepository({}, checker)
+
+    src = from_line("small-fake-a==0.1")
+    dest = copy.deepcopy(src)
+    local_repository.copy_ireq_dependencies(src, dest)
+
+    assert src in checker.copied