Repository: jazzband/pip-tools

Issue:
problems installing torch-geometric
<!-- Describe the issue briefly here. -->
` pip-sync  requirements_lock.txt ` reports missing torch for any of these packages:
```
torch-cluster==1.6.0 \
    --hash=sha256:362076bd713268ba52ed31ac9517686e206084a50bc4ea50543bdeb992d25bfe
    # via -r requirements.in
torch-geometric==2.0.4 \
    --hash=sha256:d64e4c7486fcf0c7fa82f0acbf5dd52035855469708bf89f8bc7fc607671c8b7
    # via -r requirements.in
torch-scatter==2.0.9 \
    --hash=sha256:a3e90ca7d97b0269ba5c86995f93dedd9dba80c42a65aab51d0250581ef024b4
    # via -r requirements.in
torch-sparse==0.6.14 \
    --hash=sha256:1d19c337e4842f633aedc4e23ad27c15a7ac85ce1bee0b9b016e26a821ab5f85
    # via -r requirements.in
torch-spline-conv==1.2.1 \
    --hash=sha256:959287ca87f33b8c5d901d5dccf24b1744e9c10b60fee54958fc452302059426
    # via -r requirements.in
```
#### Environment Versions

1. OS Type: Ubuntu 20.04
1. Python version: `$ python -V` = Python 3.9.5
1. pip version: `$ pip --version` = pip 22.2.1
1. pip-tools version: `$ pip-compile --version` = version 6.8.0

#### Steps to replicate
1. create requirements_lock.txt with:
```
--find-links https://download.pytorch.org/whl/torch_stable.html
--find-links https://data.pyg.org/whl/torch-1.12.0+cpu.html
torch==1.12.0+cpu \
    --hash=sha256:133023854509dd84d7ccefd99d1e9c07c392efef48c574ef6f798933c284dd6a \
    --hash=sha256:2f15f6bfcc3bca61e9b7acead9b47e6ee848c3523dedfcff3efbdd07ae6777ed \
    --hash=sha256:37f14f640b91effe41db244b932c2dd697ca2b51ae241a534259b9d9f7f51f6f \
    --hash=sha256:69afe17ea3f1bd3dd5c88ede3e275c7395463293739965e9b206f94c575d8c99 \
    --hash=sha256:6fa9e60fed5c7d7738e49e343aa1c3849ba4f3bd4da661a146d86f253e5e43cb \
    --hash=sha256:733ebe15f94edda67286b0284921b513940024d9c1943ba4122cdae4959de6e6 \
    --hash=sha256:98993de211242593626a236ee31686b2dc31b6fe564ce10a4047167b9c3eb2f5 \
    --hash=sha256:ed618d8aec7c4d46edd05fb064b2712f039b418e414a83acb959596cdf80423c
    # via
    #   -r requirements.in
    #   torchaudio
    #   torchvision
torch-cluster==1.6.0 \
    --hash=sha256:362076bd713268ba52ed31ac9517686e206084a50bc4ea50543bdeb992d25bfe
    # via -r requirements.in
torch-geometric==2.0.4 \
    --hash=sha256:d64e4c7486fcf0c7fa82f0acbf5dd52035855469708bf89f8bc7fc607671c8b7
    # via -r requirements.in
torch-scatter==2.0.9 \
    --hash=sha256:a3e90ca7d97b0269ba5c86995f93dedd9dba80c42a65aab51d0250581ef024b4
    # via -r requirements.in
torch-sparse==0.6.14 \
    --hash=sha256:1d19c337e4842f633aedc4e23ad27c15a7ac85ce1bee0b9b016e26a821ab5f85
    # via -r requirements.in
torch-spline-conv==1.2.1 \
    --hash=sha256:959287ca87f33b8c5d901d5dccf24b1744e9c10b60fee54958fc452302059426
    # via -r requirements.in
```
3.  pip-sync requirements_lock.txt

#### Expected result

pip-sync installed the packages

#### Actual result

```
Collecting torch-cluster==1.6.0
  Using cached torch_cluster-1.6.0.tar.gz (43 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-jpw__qkf/torch-cluster_04c100c202574c86820a68b9c8a6ad3f/setup.py", line 8, in <module>
          import torch
      ModuleNotFoundError: No module named 'torch'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.




```
Workaround:
```
 pip-sync --pip-args "--only-binary torch-cluster,torch-scatter,torch-sparse,torch-spline-conv" requirements_lock.txt
```
but then the hashes are different than those created by pip-compile and need to be adjusted manually... Maybe there is a way to avoid this?




Discussion:
I think an issue here is that torch-cluster doesn't use a `pyproject.toml` (or deprecated setup file stuff) to specify that torch is a build dependency, yet does in fact require torch to be pre-installed in order to build; see https://setuptools.pypa.io/en/latest/build_meta.html#dynamic-build-dependencies-and-other-build-meta-tweaks
I think you'll see the same problem installing with `pip install -r requirements_lock.txt`, as long as torch isn't preinstalled.
Another workaround I found is to install the torch deps like this:
`pip-sync --pip-args "--only-binary :all --no-binary torch-cluster,torch-scatter,torch-sparse,torch-spline-conv" requirements_lock.txt`
I think that works because the --no-binary deps will be installed last, therefore after torch is installed..?
@AndydeCleyre do you have an idea why the returned hashes might be different when using:
`pip-sync --pip-args "--only-binary torch-cluster,torch-scatter,torch-sparse,torch-spline-conv" requirements_lock.txt`
as compared to the hashes generated by pip-compile?
I double checked for example the hash for torch-scatter returned by pip-compile is correct as can be seen here:
[(https://pypi.org/project/torch-scatter/#copy-hash-modal-d4d4dc20-c5c5-4f31-9fab-4994e36cc657)](https://pypi.org/project/torch-scatter/#copy-hash-modal-d4d4dc20-c5c5-4f31-9fab-4994e36cc657):
but when running:
`pip-sync --pip-args "--only-binary torch-cluster,torch-scatter,torch-sparse,torch-spline-conv" requirements_lock.txt`
I get:
```
 (ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    torch-scatter==2.0.9 from https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_scatter-2.0.9-cp39-cp39-linux_x86_64.whl (from -r /tmp/tmpyj7b66_c (line 1)):
        Expected sha256 08f5511d64473badf0a71d156b36dc2b09b9c2f00a7cd373b935b490c477a7f1
             Got        a3e90ca7d97b0269ba5c86995f93dedd9dba80c42a65aab51d0250581ef024b4
```

Could it be that the hashes are generated based on a binary in the pypi repo instead of this one?
[https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_scatter-2.0.9-cp39-cp39-linux_x86_64.whl ](https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_scatter-2.0.9-cp39-cp39-linux_x86_64.whl )
Yeah, I'm not too familiar with the hash logic, but I think when fetching remote hashes we currently rely on the `--index-url`, maybe not accounting for `--find-links`.

Pinging @atugushev for some wisdom here

Related: #1109 
Oh, this might be an [unknown](https://github.com/jazzband/pip-tools/issues?q=is%3Aopen+is%3Aissue+label%3Ahashes) bug where pip-compile wouldn't fetch hashes from packages listed in `--find-links`. See details below.

<details>

<summary>Details</summary>

### Only a hash from PyPI
```
$ cat requirements.in
--find-links https://data.pyg.org/whl/torch-1.12.0+cpu.html
torch-cluster

$ pip-compile --generate-hashes --no-header
--find-links https://data.pyg.org/whl/torch-1.12.0+cpu.html

torch-cluster==1.6.0 \
    --hash=sha256:249c1bd8c33a887b22bf569a59d0868545804032123594dd8c76ba1885859c39
    # via -r requirements.in
```

### Hashes from `--find-link`
```
$ cat requirements.in
--no-index
--find-links https://data.pyg.org/whl/torch-1.12.0+cpu.html
torch-cluster

$ pip-compile --generate-hashes --no-header
--find-links https://data.pyg.org/whl/torch-1.12.0+cpu.html

torch-cluster==1.6.0 \
    --hash=sha256:07cc15bcad6acab550d5ffcc573a72b36e0dba477153f06d449c3ec32d7ddd9d \
    --hash=sha256:362076bd713268ba52ed31ac9517686e206084a50bc4ea50543bdeb992d25bfe \
    --hash=sha256:3a7624cae4aee89b50ea04bbccfac89c5d4e945f5000cbc975b115369dfa2942 \
    --hash=sha256:3eea6b1cec9f19e41a8478968307cd8f2405643b375d6e1a2820d842dc59474c \
    --hash=sha256:5cf2ec9100f573a73e604fa36f1e2add53ad7441b591c0beac41817626b5e3ab \
    --hash=sha256:72b48c39dce2c92e78030b99e46b6bea5fac188680aaf05a4fb973a101d86bee \
    --hash=sha256:7765a7dad5ec4876d715112065ec162e0d78465f259bdd934907468721c495f6 \
    --hash=sha256:b1698105b728b0ae8628e7d94b7201244831d72c82fed90ade092288971dd066 \
    --hash=sha256:b8e5f20186fb384008514655193e336518366e26316f3292163cc0abe969cb9c \
    --hash=sha256:e92e990d3d63035e05cacdc7d460e202afc0b56a6f6551d91bdc8b48ccf03c59 \
    --hash=sha256:ffc8b1106c0cf03c9ee5c8d2ec6f1fad4bccb0f596c8e0bae560cd2663951793
    # via -r requirements.in
```
</details>


Resolving patch:
diff --git a/piptools/repositories/pypi.py b/piptools/repositories/pypi.py
--- a/piptools/repositories/pypi.py
+++ b/piptools/repositories/pypi.py
@@ -322,21 +322,35 @@ def get_hashes(self, ireq: InstallRequirement) -> set[str]:
         log.debug(ireq.name)
 
         with log.indentation():
-            hashes = self._get_hashes_from_pypi(ireq)
-            if hashes is None:
-                log.debug("Couldn't get hashes from PyPI, fallback to hashing files")
-                return self._get_hashes_from_files(ireq)
+            return self._get_req_hashes(ireq)
 
-        return hashes
+    def _get_req_hashes(self, ireq: InstallRequirement) -> set[str]:
+        """
+        Collects the hashes for all candidates satisfying the given InstallRequirement. Computes
+        the hashes for the candidates that don't have one reported by their index.
+        """
+        matching_candidates = self._get_matching_candidates(ireq)
+        pypi_hashes_by_link = self._get_hashes_from_pypi(ireq)
+        pypi_hashes = {
+            pypi_hashes_by_link[candidate.link.url]
+            for candidate in matching_candidates
+            if candidate.link.url in pypi_hashes_by_link
+        }
+        local_hashes = {
+            self._get_file_hash(candidate.link)
+            for candidate in matching_candidates
+            if candidate.link.url not in pypi_hashes_by_link
+        }
+        return pypi_hashes | local_hashes
 
-    def _get_hashes_from_pypi(self, ireq: InstallRequirement) -> set[str] | None:
+    def _get_hashes_from_pypi(self, ireq: InstallRequirement) -> dict[str, str]:
         """
-        Return a set of hashes from PyPI JSON API for a given InstallRequirement.
-        Return None if fetching data is failed or missing digests.
+        Builds a mapping from the release URLs to their hashes as reported by the PyPI JSON API
+        for a given InstallRequirement.
         """
         project = self._get_project(ireq)
         if project is None:
-            return None
+            return {}
 
         _, version, _ = as_tuple(ireq)
 
@@ -344,23 +358,25 @@ def _get_hashes_from_pypi(self, ireq: InstallRequirement) -> set[str] | None:
             release_files = project["releases"][version]
         except KeyError:
             log.debug("Missing release files on PyPI")
-            return None
+            return {}
 
         try:
             hashes = {
-                f"{FAVORITE_HASH}:{file_['digests'][FAVORITE_HASH]}"
+                file_["url"]: f"{FAVORITE_HASH}:{file_['digests'][FAVORITE_HASH]}"
                 for file_ in release_files
                 if file_["packagetype"] in self.HASHABLE_PACKAGE_TYPES
             }
         except KeyError:
             log.debug("Missing digests of release files on PyPI")
-            return None
+            return {}
 
         return hashes
 
-    def _get_hashes_from_files(self, ireq: InstallRequirement) -> set[str]:
+    def _get_matching_candidates(
+        self, ireq: InstallRequirement
+    ) -> set[InstallationCandidate]:
         """
-        Return a set of hashes for all release files of a given InstallRequirement.
+        Returns all candidates that satisfy the given InstallRequirement.
         """
         # We need to get all of the candidates that match our current version
         # pin, these will represent all of the files that could possibly
@@ -370,11 +386,7 @@ def _get_hashes_from_files(self, ireq: InstallRequirement) -> set[str]:
         matching_versions = list(
             ireq.specifier.filter(candidate.version for candidate in all_candidates)
         )
-        matching_candidates = candidates_by_version[matching_versions[0]]
-
-        return {
-            self._get_file_hash(candidate.link) for candidate in matching_candidates
-        }
+        return candidates_by_version[matching_versions[0]]
 
     def _get_file_hash(self, link: Link) -> str:
         log.debug(f"Hashing {link.show_url}")


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import hashlib
 import os
 import shutil
 import subprocess
@@ -8,6 +9,7 @@
 from unittest import mock
 
 import pytest
+from pip._internal.utils.hashes import FAVORITE_HASH
 from pip._internal.utils.urls import path_to_url
 
 from piptools.scripts.compile import cli
@@ -1212,6 +1214,64 @@ def test_generate_hashes_with_line_style_annotations(runner):
     )
 
 
+@pytest.mark.network
+def test_generate_hashes_with_mixed_sources(
+    runner, make_package, make_wheel, make_sdist, tmp_path
+):
+    """
+    Test that pip-compile generate hashes for every file from all given sources:
+    PyPI and/or --find-links.
+    """
+
+    wheels_dir = tmp_path / "wheels"
+    wheels_dir.mkdir()
+
+    dummy_six_pkg = make_package(name="six", version="1.16.0")
+    make_wheel(dummy_six_pkg, wheels_dir, "--build-number", "123")
+
+    fav_hasher = hashlib.new(FAVORITE_HASH)
+    fav_hasher.update((wheels_dir / "six-1.16.0-123-py3-none-any.whl").read_bytes())
+    dummy_six_wheel_digest = fav_hasher.hexdigest()
+
+    with open("requirements.in", "w") as fp:
+        fp.write("six==1.16.0\n")
+
+    out = runner.invoke(
+        cli,
+        [
+            "--output-file",
+            "-",
+            "--quiet",
+            "--no-header",
+            "--generate-hashes",
+            "--no-emit-options",
+            "--no-annotate",
+            "--find-links",
+            wheels_dir.as_uri(),
+        ],
+    )
+
+    expected_digests = sorted(
+        (
+            # sdist hash for six-1.16.0.tar.gz from PyPI
+            "1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
+            # wheel hash for six-1.16.0-py2.py3-none-any.whl from PyPI
+            "8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254",
+            # wheel hash for local six-1.16.0-123-py3-none-any.whl file
+            dummy_six_wheel_digest,
+        )
+    )
+    expected_output = dedent(
+        f"""\
+        six==1.16.0 \\
+            --hash=sha256:{expected_digests[0]} \\
+            --hash=sha256:{expected_digests[1]} \\
+            --hash=sha256:{expected_digests[2]}
+        """
+    )
+    assert out.stdout == expected_output
+
+
 def test_filter_pip_markers(pip_conf, runner):
     """
     Check that pip-compile works with pip environment markers (PEP496)
diff --git a/tests/test_repository_pypi.py b/tests/test_repository_pypi.py
--- a/tests/test_repository_pypi.py
+++ b/tests/test_repository_pypi.py
@@ -4,6 +4,7 @@
 from unittest import mock
 
 import pytest
+from pip._internal.models.candidate import InstallationCandidate
 from pip._internal.models.link import Link
 from pip._internal.utils.urls import path_to_url
 from pip._vendor.requests import HTTPError, Session
@@ -177,11 +178,12 @@ def test_pip_cache_dir_is_empty(from_line, tmpdir):
                         {
                             "packagetype": "bdist_wheel",
                             "digests": {"sha256": "fake-hash"},
+                            "url": "https://link",
                         }
                     ]
                 }
             },
-            {"sha256:fake-hash"},
+            {"https://link": "sha256:fake-hash"},
             id="return single hash",
         ),
         pytest.param(
@@ -191,15 +193,20 @@ def test_pip_cache_dir_is_empty(from_line, tmpdir):
                         {
                             "packagetype": "bdist_wheel",
                             "digests": {"sha256": "fake-hash-number1"},
+                            "url": "https://link1",
                         },
                         {
                             "packagetype": "sdist",
                             "digests": {"sha256": "fake-hash-number2"},
+                            "url": "https://link2",
                         },
                     ]
                 }
             },
-            {"sha256:fake-hash-number1", "sha256:fake-hash-number2"},
+            {
+                "https://link1": "sha256:fake-hash-number1",
+                "https://link2": "sha256:fake-hash-number2",
+            },
             id="return multiple hashes",
         ),
         pytest.param(
@@ -209,39 +216,55 @@ def test_pip_cache_dir_is_empty(from_line, tmpdir):
                         {
                             "packagetype": "bdist_wheel",
                             "digests": {"sha256": "fake-hash-number1"},
+                            "url": "https://link1",
                         },
                         {
                             "packagetype": "sdist",
                             "digests": {"sha256": "fake-hash-number2"},
+                            "url": "https://link2",
                         },
                         {
                             "packagetype": "bdist_eggs",
                             "digests": {"sha256": "fake-hash-number3"},
+                            "url": "https://link3",
                         },
                     ]
                 }
             },
-            {"sha256:fake-hash-number1", "sha256:fake-hash-number2"},
+            {
+                "https://link1": "sha256:fake-hash-number1",
+                "https://link2": "sha256:fake-hash-number2",
+            },
             id="return only bdist_wheel and sdist hashes",
         ),
-        pytest.param(None, None, id="not found project data"),
-        pytest.param({}, None, id="not found releases key"),
-        pytest.param({"releases": {}}, None, id="not found version"),
-        pytest.param({"releases": {"0.1": [{}]}}, None, id="not found digests"),
+        pytest.param(None, {}, id="not found project data"),
+        pytest.param({}, {}, id="not found releases key"),
+        pytest.param({"releases": {}}, {}, id="not found version"),
+        pytest.param({"releases": {"0.1": [{}]}}, {}, id="not found digests"),
         pytest.param(
-            {"releases": {"0.1": [{"packagetype": "bdist_wheel", "digests": {}}]}},
-            None,
+            {
+                "releases": {
+                    "0.1": [
+                        {"packagetype": "bdist_wheel", "digests": {}, "url": "link"}
+                    ]
+                }
+            },
+            {},
             id="digests are empty",
         ),
         pytest.param(
             {
                 "releases": {
                     "0.1": [
-                        {"packagetype": "bdist_wheel", "digests": {"md5": "fake-hash"}}
+                        {
+                            "packagetype": "bdist_wheel",
+                            "digests": {"md5": "fake-hash"},
+                            "url": "https://link",
+                        }
                     ]
                 }
             },
-            None,
+            {},
             id="not found sha256 algo",
         ),
     ),
@@ -264,6 +287,64 @@ def _get_project(self, ireq):
     assert actual_hashes == expected_hashes
 
 
+def test_get_hashes_from_mixed(pip_conf, from_line, tmpdir):
+    """
+    Test PyPIRepository.get_hashes() returns hashes from both PyPi and extra indexes/links
+    """
+
+    package_name = "small-fake-multi-arch"
+    package_version = "0.1"
+
+    # One candidate from PyPi and the rest from find-links / extra indexes
+    extra_index_link1 = Link("https://extra-index-link1")
+    extra_index_link2 = Link("https://extra-index-link2")
+    pypi_link = Link("https://pypi-link")
+
+    all_candidates = [
+        InstallationCandidate(package_name, package_version, extra_index_link1),
+        InstallationCandidate(package_name, package_version, extra_index_link2),
+        InstallationCandidate(package_name, package_version, pypi_link),
+    ]
+
+    # Extra indexes hashes so we don't spend time computing them
+    file_hashes = {
+        extra_index_link1: "sha256:hash-link1",
+        extra_index_link2: "sha256:hash-link2",
+    }
+    pypi_hash = "pypi-hash"
+
+    class MockPyPIRepository(PyPIRepository):
+        def _get_project(self, ireq):
+            return {
+                "releases": {
+                    package_version: [
+                        {
+                            "packagetype": "bdist_wheel",
+                            "digests": {"sha256": pypi_hash},
+                            "url": str(pypi_link),
+                        },
+                    ]
+                }
+            }
+
+        def find_all_candidates(self, req_name):
+            return all_candidates
+
+        def _get_file_hash(self, link):
+            return file_hashes[link]
+
+    pypi_repository = MockPyPIRepository(
+        ["--no-cache-dir"], cache_dir=(tmpdir / "pypi-repo-cache")
+    )
+
+    ireq = from_line(f"{package_name}=={package_version}")
+
+    expected_hashes = {"sha256:" + pypi_hash} | set(file_hashes.values())
+
+    actual_hashes = pypi_repository.get_hashes(ireq)
+    assert actual_hashes == expected_hashes
+
+
 def test_get_project__returns_data(from_line, tmpdir, monkeypatch, pypi_repository):
     """
     Test PyPIRepository._get_project() returns expected project data.