Repository: jazzband/pip-tools

Issue:
Possible bug: problems resolving pytorch (cuda).
This may very well be PyTorch issue due to the way they distribute packages, but I will describe it here too.
The problem is that `pip-compile` doesn't stick to the way how package version is descried in `.in` file.
I am trying to install the default version of PyTorch (which comes compiled against CUDA 10.1).

#### Environment Versions

1. OS Type: Windows 10
1. Python version: 3.7.6
1. pip version:  20.0.2 
1. pip-tools version: 4.5.1

#### Steps to replicate

1. Make `requirements.in` file:
    ```
    torch===1.4.0
    torchvision===0.5.0
    ```
    Note usage of [arbitrary equality](https://www.python.org/dev/peps/pep-0440/#arbitrary-equality). I know that this is discouraged by pip. This is way I think the issue may be more related to PyTorch itself.
2. Compile it `pip-compile -f https://download.pytorch.org/whl/torch_stable.html -o requirements.txt requirements.in`

#### Expected result

Expected `requirements.txt` with `torch===1.4.0` (note triple `=`).

#### Actual result

Actual result it:
```
torch==1.4.0              # via -r req.in, torchvision
torchvision===0.5.0       # via -r req.in
```
Note double `=` on torch and triple `=` on torchvision.

If one installs the packages via `pip-sync requirements.txt`, then:
```
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting numpy==1.18.3
  Using cached numpy-1.18.3-cp37-cp37m-win_amd64.whl (12.8 MB)
Collecting pillow==7.1.1
  Using cached Pillow-7.1.1-cp37-cp37m-win_amd64.whl (2.0 MB)
Collecting torch==1.4.0
  Using cached https://download.pytorch.org/whl/cu92/torch-1.4.0%2Bcu92-cp37-cp37m-win_amd64.whl (641.8 MB)
Collecting torchvision===0.5.0
  Using cached torchvision-0.5.0-cp37-cp37m-win_amd64.whl (1.2 MB)
```

See that `cu92` version of `torch` is selected.

However, if `requirements.txt` would contain `torch===1.4.0`, then the results of `pip-sync` would be:
```
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch===1.4.0
  Using cached https://download.pytorch.org/whl/cu101/torch-1.4.0-cp37-cp37m-win_amd64.whl (796.8 MB)
Collecting torchvision===0.5.0
  Using cached torchvision-0.5.0-cp37-cp37m-win_amd64.whl (1.2 MB)
Successfully installed torch-1.4.0 torchvision-0.5.0
```




Discussion:
Hello @fralik,

Thanks for the issue! Semantically `===` and `==` [are equal](https://github.com/pypa/pip/blob/814c54f5d7e17682f9c476f5978c4d5666abb52c/src/pip/_vendor/distlib/version.py#L89). TBH I have no idea why pip chooses different wheels depending on version matcher specifier. However, I cannot reproduce on macOS, see:

#### With `==`
```console
$ cat requirements.txt
torch==1.4.0

$ pip-sync
Collecting torch==1.4.0
  Downloading torch-1.4.0-cp38-none-macosx_10_9_x86_64.whl (81.2 MB)
```

#### With `===`
```console
$ cat requirements.txt
torch==1.4.0

$ pip-sync
Collecting torch===1.4.0
  Downloading torch-1.4.0-cp38-none-macosx_10_9_x86_64.whl (81.2 MB)
```

UPDATE: ah, sorry, I should use `--find-links=https://download.pytorch.org/whl/torch_stable.html`, but it looks like it doesn't reproduce because of there are no `cpu` wheels for macOS.

UPDATE 2: can confirm on Linux with docker.
A stacktrace for curious ones:

```console
user@home:/$ docker run --rm -it python:3.8-slim /bin/bash
root@f8514dadee50:/# pip install torch==1.4.0 --find-links https://download.pytorch.org/whl/torch_stable.html
```
```traceback
[0]   /usr/local/bin/pip(8)<module>()
-> sys.exit(main())
[1]   /usr/local/lib/python3.8/site-packages/pip/_internal/cli/main.py(75)main()
-> return command.main(cmd_args)
[2]   /usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py(105)main()
-> return self._main(args)
[3]   /usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py(186)_main()
-> status = self.run(options, args)
[4]   /usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py(331)run()
-> resolver.resolve(requirement_set)
[5]   /usr/local/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py(177)resolve()
-> discovered_reqs.extend(self._resolve_one(requirement_set, req))
[6]   /usr/local/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py(333)_resolve_one()
-> abstract_dist = self._get_abstract_dist_for(req_to_install)
[7]   /usr/local/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py(281)_get_abstract_dist_for()
-> req.populate_link(self.finder, upgrade_allowed, require_hashes)
[8]   /usr/local/lib/python3.8/site-packages/pip/_internal/req/req_install.py(249)populate_link()
-> self.link = finder.find_requirement(self, upgrade)
[9]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(898)find_requirement()
-> best_candidate_result = self.find_best_candidate(
[10]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(887)find_best_candidate()
-> return candidate_evaluator.compute_best_candidate(candidates)
[11]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(584)compute_best_candidate()
-> applicable_candidates = self.get_applicable_candidates(candidates)
[12]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(459)get_applicable_candidates()
-> versions = {
[13]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(459)<setcomp>()
-> versions = {
[14]   /usr/local/lib/python3.8/site-packages/pip/_vendor/packaging/specifiers.py(237)filter()
-> if self.contains(parsed_version, **kw):
[15]   /usr/local/lib/python3.8/site-packages/pip/_vendor/packaging/specifiers.py(221)contains()
-> return operator_callable(normalized_item, version)
[16]   /usr/local/lib/python3.8/site-packages/pip/_vendor/packaging/specifiers.py(324)wrapped()
-> return fn(self, prospective, spec)
[17] > /usr/local/lib/python3.8/site-packages/pip/_vendor/packaging/specifiers.py(503)_compare_equal()
-> prospective = Version(prospective.public)
(Pdb++)
```

```python
>>> print(self)
<Specifier('==1.4.0')>

>>> print(prospective)
<Version('1.4.0+cpu')>

>>> print(prospective.public)
'1.4.0'
```
PEP-440 about arbitrary `===` operator https://www.python.org/dev/peps/pep-0440/#arbitrary-equality
Could be related also to #785, since scm dev versions have the same notation like `1.0.1.dev3+gd7cef3b`.
Any news on this ? I have the same issue where I need to force the torch version with === and the generated .txt file only contains == which does not seem to enforce the proper version when using pip-sync

Cheers
Hello @IceTDrinker,

Thanks for pinging. AFAIK, nobody works on the fix. Any help is highly appreciated and welcomed!
A pointer as to where I should look ? 🙂
@IceTDrinker 

Probably, this could be a good start:
https://github.com/jazzband/pip-tools/blob/f60fb001103ec2c63ce7540acb12da5d822c60de/piptools/utils.py#L78

Note hardcoded `==`. Perhaps, there should be some way to resolve `===` to `===`.
Looks like a good place to start indeed. I'll probably take a look this week-end thanks !

Resolving patch:
diff --git a/piptools/repositories/local.py b/piptools/repositories/local.py
--- a/piptools/repositories/local.py
+++ b/piptools/repositories/local.py
@@ -63,9 +63,7 @@ def find_best_match(self, ireq, prereleases=None):
         existing_pin = self.existing_pins.get(key)
         if existing_pin and ireq_satisfied_by_existing_pin(ireq, existing_pin):
             project, version, _ = as_tuple(existing_pin)
-            return make_install_requirement(
-                project, version, ireq.extras, constraint=ireq.constraint
-            )
+            return make_install_requirement(project, version, ireq)
         else:
             return self.repository.find_best_match(ireq, prereleases)
 
diff --git a/piptools/repositories/pypi.py b/piptools/repositories/pypi.py
--- a/piptools/repositories/pypi.py
+++ b/piptools/repositories/pypi.py
@@ -159,8 +159,7 @@ def find_best_match(self, ireq, prereleases=None):
         return make_install_requirement(
             best_candidate.name,
             best_candidate.version,
-            ireq.extras,
-            constraint=ireq.constraint,
+            ireq,
         )
 
     def resolve_reqs(self, download_dir, ireq, wheel_cache):
diff --git a/piptools/utils.py b/piptools/utils.py
--- a/piptools/utils.py
+++ b/piptools/utils.py
@@ -21,6 +21,7 @@
 from pip._internal.vcs import is_url
 from pip._vendor.packaging.markers import Marker
 from pip._vendor.packaging.specifiers import SpecifierSet
+from pip._vendor.packaging.version import Version
 
 _KT = TypeVar("_KT")
 _VT = TypeVar("_VT")
@@ -66,16 +67,25 @@ def comment(text: str) -> str:
 
 
 def make_install_requirement(
-    name: str, version: str, extras: Iterable[str], constraint: bool = False
+    name: str, version: Union[str, Version], ireq: InstallRequirement
 ) -> InstallRequirement:
     # If no extras are specified, the extras string is blank
     extras_string = ""
+    extras = ireq.extras
     if extras:
         # Sort extras for stability
         extras_string = f"[{','.join(sorted(extras))}]"
 
+    version_pin_operator = "=="
+    version_as_str = str(version)
+    for specifier in ireq.specifier:
+        if specifier.operator == "===" and specifier.version == version_as_str:
+            version_pin_operator = "==="
+            break
+
     return install_req_from_line(
-        str(f"{name}{extras_string}=={version}"), constraint=constraint
+        str(f"{name}{extras_string}{version_pin_operator}{version}"),
+        constraint=ireq.constraint,
     )
 
 


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -68,9 +68,7 @@ def find_best_match(self, ireq, prereleases=False):
             ]
             raise NoCandidateFound(ireq, tried_versions, ["https://fake.url.foo"])
         best_version = max(versions, key=Version)
-        return make_install_requirement(
-            key_from_ireq(ireq), best_version, ireq.extras, constraint=ireq.constraint
-        )
+        return make_install_requirement(key_from_ireq(ireq), best_version, ireq)
 
     def get_dependencies(self, ireq):
         if ireq.editable or is_url_requirement(ireq):
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -1555,3 +1555,74 @@ def test_duplicate_reqs_combined(
     assert out.exit_code == 0, out
     assert str(test_package_2) in out.stderr
     assert "test-package-1==0.1" in out.stderr
+
+
+@pytest.mark.parametrize(
+    ("pkg2_install_requires", "req_in_content", "out_expected_content"),
+    (
+        pytest.param(
+            "",
+            ["test-package-1===0.1.0\n"],
+            ["test-package-1===0.1.0"],
+            id="pin package with ===",
+        ),
+        pytest.param(
+            "",
+            ["test-package-1==0.1.0\n"],
+            ["test-package-1==0.1.0"],
+            id="pin package with ==",
+        ),
+        pytest.param(
+            "test-package-1==0.1.0",
+            ["test-package-1===0.1.0\n", "test-package-2==0.1.0\n"],
+            ["test-package-1===0.1.0", "test-package-2==0.1.0"],
+            id="dep === pin preferred over == pin, main package == pin",
+        ),
+        pytest.param(
+            "test-package-1==0.1.0",
+            ["test-package-1===0.1.0\n", "test-package-2===0.1.0\n"],
+            ["test-package-1===0.1.0", "test-package-2===0.1.0"],
+            id="dep === pin preferred over == pin, main package === pin",
+        ),
+        pytest.param(
+            "test-package-1==0.1.0",
+            ["test-package-2===0.1.0\n"],
+            ["test-package-1==0.1.0", "test-package-2===0.1.0"],
+            id="dep == pin conserved, main package === pin",
+        ),
+    ),
+)
+def test_triple_equal_pinned_dependency_is_used(
+    runner,
+    make_package,
+    make_wheel,
+    tmpdir,
+    pkg2_install_requires,
+    req_in_content,
+    out_expected_content,
+):
+    """
+    Test that pip-compile properly emits the pinned requirement with ===
+    torchvision 0.8.2 requires torch==1.7.1 which can resolve to versions with
+    patches (e.g. torch 1.7.1+cu110), we want torch===1.7.1 without patches
+    """
+
+    dists_dir = tmpdir / "dists"
+
+    test_package_1 = make_package("test_package_1", version="0.1.0")
+    make_wheel(test_package_1, dists_dir)
+
+    test_package_2 = make_package(
+        "test_package_2", version="0.1.0", install_requires=[pkg2_install_requires]
+    )
+    make_wheel(test_package_2, dists_dir)
+
+    with open("requirements.in", "w") as reqs_in:
+        for line in req_in_content:
+            reqs_in.write(line)
+
+    out = runner.invoke(cli, ["--find-links", str(dists_dir)])
+
+    assert out.exit_code == 0, out
+    for line in out_expected_content:
+        assert line in out.stderr