Repository: jazzband/pip-tools

Issue:
Multiple entries in requirements.in with the same file cause collision
When a requirements.in contains multiple entries with the same file name, there is a collision when they are downloaded. e.g.

requirements.in contents:
`https://github.com/cloudify-cosmo/cloudify-common/archive/master.zip#egg=cloudify-common[dispatcher]`
`https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip#egg=cloudify`

This results in a warning that the cloudify 5.1.0.dev1 does not provide the extra 'dispatcher',
followed by an error that no version could be found for cloudify-common==5.1.0.dev1

#### Environment Versions

1. Ubuntu 18.04
1. Python version: Python 2.7.17  # Though by the behaviour I expect this to still cause problems when we switch this to Python 3.
1. pip version:  pip 20.1
1. pip-tools version: pip-compile, version 5.1.2

#### Steps to replicate

1. Create an empty dir, and cd to it
2. Make a requirements.in file in this dir, containing:
`https://github.com/cloudify-cosmo/cloudify-common/archive/master.zip#egg=cloudify-common[dispatcher]`
`https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip#egg=cloudify`
3. Run `pip-compile`

#### Expected result

requirements.txt is produced.

#### Actual result

Errors abound (see above for specific errors).


Discussion:
Hello @geokala, 

Thanks for the issue! This problem is not related to how many entries with the same filenames in `requirements.in`. Actually, the issue is with [setup.py](https://github.com/cloudify-cosmo/cloudify-cli/blob/29abf556829d67eb75821dd056af41849f6bc628/setup.py#L51) in `cloudify-cli`. There `cloudify-common` pinned to `5.1.0.dev1`, which is not uploaded to PyPI. See the reproducer and error message:

```console
$ cat requirements.in
https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip#egg=cloudify

$ pip-compile
```

> ERROR: Could not find a version that satisfies the requirement **cloudify-common==5.1.0.dev1** (from cloudify->-r requirements.in (line 2)) (from versions: 4.4, 4.5, 4.5.5, 4.6, 5.0.0, 5.0.5.1)


I'd suggest to open issue on `cloudify-cli`'s issue tracker.
Hi @atugushev 
There is indeed a pinning of that version, but that is provided in the first line.

The dependencies work just fine if you put them in a requirements.txt file and pip install -r requirements.txt

They're failing to work when used with pip-compile (possibly due to one of the tools being used by it under the hood), probably because there's a cached master.zip being overwritten (I think- from discussion with a coworker- haven't checked directly myself).

Based on that, I don't believe this bug is invalid.

(I'm actually working on cloudify and we're using this for some pre-release testing.)

Thanks,
S
Yes, indeed. Sorry, I've overlooked that fact. I've compiled with `--verbose` flag and noticed the warning:

```
Finding secondary dependencies:
  Collecting cloudify
    Using cached https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip
    Saved /root/.cache/pip-tools/pkgs/master.zip
  Collecting cloudify-common
    File was already downloaded /root/.cache/pip-tools/pkgs/master.zip
    WARNING: Generating metadata for package cloudify-common produced metadata for project name cloudify. Fix your #egg=cloudify-common fragments.
```

There must've been a bug with conflicted cache names. I made an experiment and it's confirmed your suspicious:

```console
$ cd /tmp

$ wget https://github.com/cloudify-cosmo/cloudify-common/archive/master.zip -O cloudify-common-master.zip

$ wget https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip -O cloudify-cli-master.zip

$ cat requirements.in
file:///tmp/cloudify-common-master.zip#egg=cloudify-common[dispatcher]
file:///tmp/cloudify-cli-master.zip#egg=cloudify

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
appdirs==1.4.4            # via virtualenv
backports.shutil_get_terminal_size==1.0.0  # via cloudify
bottle==0.12.18           # via cloudify-common
certifi==2020.4.5.1       # via requests
chardet==3.0.4            # via requests
click==7.1.2              # via click-didyoumean, cloudify
click_didyoumean==0.0.3   # via cloudify
file:///tmp/cloudify-cli-master.zip#egg=cloudify  # via -r requirements.in
file:///tmp/cloudify-common-master.zip#egg=cloudify-common[dispatcher]  # via -r requirements.in, cloudify
colorama==0.3.3           # via cloudify
configparser==4.0.2       # via importlib-metadata
contextlib2==0.6.0.post1  # via importlib-metadata, importlib-resources, virtualenv
decorator==4.4.2          # via networkx
distlib==0.3.0            # via virtualenv
filelock==3.0.12          # via virtualenv
idna==2.9                 # via requests
importlib-metadata==1.6.0  # via importlib-resources, virtualenv
importlib-resources==1.5.0  # via virtualenv
ipaddress==1.0.19         # via cloudify
jinja2==2.10              # via cloudify, cloudify-common
markupsafe==1.1.1         # via jinja2
networkx==1.9.1           # via cloudify-common
pathlib2==2.3.5           # via importlib-metadata, importlib-resources, virtualenv
pika==0.11.2              # via cloudify-common
ply==3.11                 # via pysmi
prettytable==0.7.2        # via cloudify
proxy_tools==0.1.0        # via cloudify-common
pyasn1==0.4.8             # via pysnmp
pycryptodomex==3.9.7      # via pysnmp
pysmi==0.3.4              # via pysnmp
pysnmp==4.4.5             # via cloudify-common
pyyaml==3.12              # via cloudify, cloudify-common
requests==2.23.0          # via cloudify, cloudify-common, requests-toolbelt
requests_toolbelt==0.8.0  # via cloudify-common
retrying==1.3.3           # via cloudify, cloudify-common
scandir==1.10.0           # via pathlib2
singledispatch==3.4.0.3   # via importlib-resources
six==1.14.0               # via pathlib2, retrying, virtualenv
typing==3.7.4.1           # via importlib-resources
urllib3==1.25.9           # via requests
virtualenv==20.0.20       # via wagon
wagon[venv]==0.6.3        # via cloudify
wheel==0.29.0             # via wagon
zipp==1.2.0               # via importlib-metadata, importlib-resources

# The following packages are considered to be unsafe in a requirements file:
# setuptools
```


Thanks for the good catch @geokala! 👍 
No worries.

Do you have any rough idea of where the root cause might be? I might (depending on other ongoing stuff) try to take a dig to fix it myself.
Sure! This download dir is global for all links: https://github.com/jazzband/pip-tools/blob/b1d73691e9064467326ae915a2753670de643925/piptools/repositories/pypi.py#L86

There `pip-tools` passes this dir to `pip`:
https://github.com/jazzband/pip-tools/blob/b1d73691e9064467326ae915a2753670de643925/piptools/repositories/pypi.py#L161

The idea might be to add to the download directory (before it passed to `make_requirement_preparer`) a salt based on a link URL to avoid collision, for example:
```python
salt = hashlib.md5(ireq.link.url_without_fragment.encode()).hexdigest()
donwload_dir=os.path.join(download_dir, salt[:2], salt[2:4], salt[4:6], ...)
```
Sounds good, I'll try to have a look at it one evening this week!

Resolving patch:
diff --git a/piptools/repositories/pypi.py b/piptools/repositories/pypi.py
--- a/piptools/repositories/pypi.py
+++ b/piptools/repositories/pypi.py
@@ -215,7 +215,7 @@ def get_dependencies(self, ireq):
                 # using git-checkout-index, which gets rid of the .git dir.
                 download_dir = None
             else:
-                download_dir = self._download_dir
+                download_dir = self._get_download_path(ireq)
                 if not os.path.isdir(download_dir):
                     os.makedirs(download_dir)
             if not os.path.isdir(self._wheel_download_dir):
@@ -288,6 +288,20 @@ def _get_project(self, ireq):
             return data
         return None
 
+    def _get_download_path(self, ireq):
+        """
+        Determine the download dir location in a way which avoids name
+        collisions.
+        """
+        if ireq.link:
+            salt = hashlib.sha224(ireq.link.url_without_fragment.encode()).hexdigest()
+            # Nest directories to avoid running out of top level dirs on some FS
+            # (see pypi _get_cache_path_parts, which inspired this)
+            salt = [salt[:2], salt[2:4], salt[4:6], salt[6:]]
+            return os.path.join(self._download_dir, *salt)
+        else:
+            return self._download_dir
+
     def get_hashes(self, ireq):
         """
         Given an InstallRequirement, return a set of hashes that represent all
@@ -308,7 +322,7 @@ def get_hashes(self, ireq):
                 # Directly hash URL requirements.
                 # URL requirements may have been previously downloaded and cached
                 # locally by self.resolve_reqs()
-                cached_path = os.path.join(self._download_dir, link.filename)
+                cached_path = os.path.join(self._get_download_path(ireq), link.filename)
                 if os.path.exists(cached_path):
                     cached_link = Link(path_to_url(cached_path))
                 else:


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -271,11 +271,13 @@ def run_setup_file():
     Run a setup.py file from a given package dir.
     """
 
-    def _make_wheel(package_dir_path, *args):
+    def _run_setup_file(package_dir_path, *args):
         setup_file = str(package_dir_path / "setup.py")
-        return check_call((sys.executable, setup_file) + args)  # nosec
+        return check_call(
+            (sys.executable, setup_file) + args, cwd=str(package_dir_path)
+        )  # nosec
 
-    return _make_wheel
+    return _run_setup_file
 
 
 @pytest.fixture
diff --git a/tests/test_repository_pypi.py b/tests/test_repository_pypi.py
--- a/tests/test_repository_pypi.py
+++ b/tests/test_repository_pypi.py
@@ -347,3 +347,45 @@ def mock_get(*args, **kwargs):
 
     actual_data = pypi_repository._get_project(ireq)
     assert actual_data is None
+
+
+def test_name_collision(from_line, pypi_repository, make_package, make_sdist, tmpdir):
+    """
+    Test to ensure we don't fail if there are multiple URL-based requirements
+    ending with the same filename where later ones depend on earlier, e.g.
+    https://git.example.com/requirement1/master.zip#egg=req_package_1
+    https://git.example.com/requirement2/master.zip#egg=req_package_2
+    In this case, if req_package_2 depends on req_package_1 we don't want to
+    fail due to issues such as caching the requirement based on filename.
+    """
+    packages = {
+        "test_package_1": make_package("test_package_1", version="0.1"),
+        "test_package_2": make_package(
+            "test_package_2", version="0.1", install_requires=["test-package-1"]
+        ),
+    }
+
+    for pkg_name, pkg in packages.items():
+        pkg_path = str(tmpdir / pkg_name)
+
+        make_sdist(pkg, pkg_path, "--formats=zip")
+
+        os.rename(
+            os.path.join(pkg_path, "{}-{}.zip".format(pkg_name, "0.1")),
+            os.path.join(pkg_path, "master.zip"),
+        )
+
+    name_collision_1 = "file://{dist_path}#egg=test_package_1".format(
+        dist_path=tmpdir / "test_package_1" / "master.zip"
+    )
+    ireq = from_line(name_collision_1)
+    deps = pypi_repository.get_dependencies(ireq)
+    assert len(deps) == 0
+
+    name_collision_2 = "file://{dist_path}#egg=test_package_2".format(
+        dist_path=tmpdir / "test_package_2" / "master.zip"
+    )
+    ireq = from_line(name_collision_2)
+    deps = pypi_repository.get_dependencies(ireq)
+    assert len(deps) == 1
+    assert deps.pop().name == "test-package-1"