Repository: pex-tool/pex

Issue:
update on lock with acryl-datahub fails with cryptic error message
```
$ python3.10 -m pex.cli lock create '--indent=2'  --manylinux manylinux2014 --interpreter-constraint $'CPython==3.10.*' --style=universal --pip-version=23.3.2 --resolver-version pip-2020-resolver --target-system=linux --target-system=mac  --prefer-binary -o tmp/ad-lock.json acryl-datahub==0.12.0.5
```

```
$ pex3 lock update '--indent=2'  --manylinux manylinux2014  --project=pydot tmp/ad-lock.json
...
Encountered 1 error updating tmp/ad-lock.json:
1.) cp311-cp311-manylinux_2_37_x86_64: 
```


To get some more details with `PEX_VERBOSE=10`:
```
Traceback (most recent call last):
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/resolve/lockfile/updater.py", line 320, in update_resolve
    assert updated_requirement.artifact == locked_requirement.artifact
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Encountered 1 error updating tmp/ad-lock.json:
1.) cp311-cp311-manylinux_2_37_x86_64: 
```

With some dinosaur `print` debugging:
```
                assert updated_pin == original_pin
                if updated_requirement.artifact != locked_requirement.artifact:
                    print('up ',  updated_requirement, 'lock', locked_requirement)
                assert updated_requirement.artifact == locked_requirement.artifact
```


```
up  LockedRequirement(pin=Pin(project_name=ProjectName(raw='avro', normalized='avro'), version=Version(raw='1.11.3', normalized='1.11.3')), artifact=FileArtifact(url='https://files.pythonhosted.org/packages/58/5b/41d2dab437adbad4077acba55b05064c5eb0aea8e77145a0379564d32950/avro-1.11.3.tar.gz', fingerprint=Fingerprint(algorithm='sha256', hash='3393bb5139f9cf0791d205756ce1e39a5b58586af5b153d6a3b5a199610e9d17'), verified=True, filename='avro-1.11.3.tar.gz'), requires_dists=(Requirement(name='python-snappy', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('extra == "snappy"')>), Requirement(name='typing-extensions', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('python_version < "3.8"')>), Requirement(name='zstandard', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('extra == "zstandard"')>)), requires_python=<SpecifierSet('>=3.6')>, additional_artifacts=()) lock LockedRequirement(pin=Pin(project_name=ProjectName(raw='avro', normalized='avro'), version=Version(raw='1.11.3', normalized='1.11.3')), artifact=FileArtifact(url='https://files.pythonhosted.org/packages/58/5b/41d2dab437adbad4077acba55b05064c5eb0aea8e77145a0379564d32950/avro-1.11.3.tar.gz', fingerprint=Fingerprint(algorithm='sha256', hash='3393bb5139f9cf0791d205756ce1e39a5b58586af5b153d6a3b5a199610e9d17'), verified=False, filename='avro-1.11.3.tar.gz'), requires_dists=(Requirement(name='python-snappy', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('extra == "snappy"')>), Requirement(name='typing-extensions', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('python_version < "3.8"')>), Requirement(name='zstandard', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=<Marker('extra == "zstandard"')>)), requires_python=<SpecifierSet('>=3.6')>, additional_artifacts=())
```

Which I believe simplifies to:
```
$ diff -u tmp/up tmp/lock
--- tmp/up	2024-01-12 10:17:17.372343836 -0500
+++ tmp/lock	2024-01-12 10:17:25.575630776 -0500
@@ -5,7 +5,7 @@
 artifact=FileArtifact(url='https://files.pythonhosted.org/packages/58/5b/41d2dab437adbad4077acba55b05064c5eb0aea8e77145a0379564d32950/avro-1.11.3.tar.gz', 
 fingerprint=Fingerprint(algorithm='sha256', 
 hash='3393bb5139f9cf0791d205756ce1e39a5b58586af5b153d6a3b5a199610e9d17'), 
-verified=True, 
+verified=False, 
 filename='avro-1.11.3.tar.gz'), 
 requires_dists=(Requirement(name='python-snappy', 
 url=None, 
```

Which I think shows why the assertion was triggered, but I still don't see what about this package is tickling the `verified` 

 * Pex versions tried: 2.1.152, 2.1.156, and `main`
 * `avro==1.11.3` instead of `acryl-datahub==0.12.0.5` in the original `create` also fails in the same way.


Discussion:
@cburroughs I won't have time to work on this until tomorrow afternoon. The verified flag indicates whether the artifact hash has been calculated by Pex itself (`verified=True`), or taken as given (`verified=False`); i.e.: taken from a PyPI index artifact URL or JSON API metadata.
Noting this fixes:
```diff
$ git diff
diff --git a/pex/resolve/locked_resolve.py b/pex/resolve/locked_resolve.py
index 241cb837..c4421063 100644
--- a/pex/resolve/locked_resolve.py
+++ b/pex/resolve/locked_resolve.py
@@ -148,7 +148,7 @@ class Artifact(object):

     url = attr.ib()  # type: str
     fingerprint = attr.ib()  # type: Fingerprint
-    verified = attr.ib()  # type: bool
+    verified = attr.ib(eq=False)  # type: bool

     def __lt__(self, other):
         # type: (Any) -> bool
```

An integration test is needed to prevent backslide though as well as a note explaining why this is right to ignore in `__eq__`.

Resolving patch:
diff --git a/pex/resolve/lockfile/updater.py b/pex/resolve/lockfile/updater.py
--- a/pex/resolve/lockfile/updater.py
+++ b/pex/resolve/lockfile/updater.py
@@ -34,7 +34,7 @@
 from pex.util import named_temporary_file
 
 if TYPE_CHECKING:
-    from typing import Dict, Iterable, Iterator, List, Mapping, Optional, Tuple, Union
+    from typing import Any, Dict, Iterable, Iterator, List, Mapping, Optional, Tuple, Union
 
     import attr  # vendor:skip
 else:
@@ -316,11 +316,56 @@ def update_resolve(
                 self.update_constraints_by_project_name
                 and project_name not in self.update_constraints_by_project_name
             ):
-                assert updated_pin == original_pin
-                assert updated_requirement.artifact == locked_requirement.artifact
-                assert (
-                    updated_requirement.additional_artifacts
-                    == locked_requirement.additional_artifacts
+                assert updated_pin == original_pin, (
+                    "The locked requirement {original} should have been undisturbed by the lock "
+                    "update, but it changed to {updated}.".format(
+                        original=original_pin.as_requirement(), updated=updated_pin.as_requirement()
+                    )
+                )
+
+                # N.B.: We use a custom key for artifact equality comparison since `Artifact`
+                # contains a `verified` attribute that can both vary based on Pex's current
+                # knowledge about the trustworthiness of an artifact hash and is not relevant to
+                # whether the artifact refers to the same artifact.
+                def artifact_key(artifact):
+                    # type: (Artifact) -> Any
+                    return artifact.url, artifact.fingerprint
+
+                assert artifact_key(updated_requirement.artifact) == artifact_key(
+                    locked_requirement.artifact
+                ), (
+                    "The locked requirement {original} should have been undisturbed by the lock "
+                    "update, but its primary artifact changed from:\n"
+                    "{original_artifact}\n"
+                    "to:\n"
+                    "{updated_artifact}".format(
+                        original=original_pin.as_requirement(),
+                        original_artifact=artifact_key(locked_requirement.artifact),
+                        updated_artifact=artifact_key(updated_requirement.artifact),
+                    )
+                )
+                assert set(map(artifact_key, updated_requirement.additional_artifacts)) == set(
+                    map(artifact_key, locked_requirement.additional_artifacts)
+                ), (
+                    "The locked requirement {original} should have been undisturbed by the lock "
+                    "update, but its additional artifact set changed from:\n"
+                    "{original_artifacts}\n"
+                    "to:\n"
+                    "{updated_artifacts}".format(
+                        original=original_pin.as_requirement(),
+                        original_artifacts="\n".join(
+                            map(
+                                lambda a: str(artifact_key(a)),
+                                locked_requirement.additional_artifacts,
+                            )
+                        ),
+                        updated_artifacts="\n".join(
+                            map(
+                                lambda a: str(artifact_key(a)),
+                                updated_requirement.additional_artifacts,
+                            )
+                        ),
+                    )
                 )
             elif original_pin != updated_pin:
                 updates[project_name] = VersionUpdate(
diff --git a/pex/version.py b/pex/version.py
--- a/pex/version.py
+++ b/pex/version.py
@@ -1,4 +1,4 @@
 # Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
-__version__ = "2.1.156"
+__version__ = "2.1.157"


Test patch:
diff --git a/tests/integration/test_issue_2324.py b/tests/integration/test_issue_2324.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/test_issue_2324.py
@@ -0,0 +1,80 @@
+# Copyright 2023 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+from __future__ import print_function
+
+import itertools
+import os.path
+
+from pex.pep_440 import Version
+from pex.pep_503 import ProjectName
+from pex.resolve.locked_resolve import FileArtifact
+from pex.resolve.lockfile import json_codec
+from pex.resolve.resolved_requirement import Pin
+from pex.typing import TYPE_CHECKING
+from testing.cli import run_pex3
+
+if TYPE_CHECKING:
+    from typing import Any
+
+
+def test_update_sdists_not_updated(tmpdir):
+    # type: (Any) -> None
+
+    constraints = os.path.join(str(tmpdir), "constraints.txt")
+    with open(constraints, "w") as fp:
+        print("ansicolors<1.1.8", file=fp)
+        print("cowsay<6", file=fp)
+
+    lock = os.path.join(str(tmpdir), "lock.json")
+
+    def assert_lock(*pins):
+        # type: (*Pin) -> None
+
+        lockfile = json_codec.load(lock)
+        assert 1 == len(lockfile.locked_resolves)
+        locked_resolve = lockfile.locked_resolves[0]
+        locked_requirements = {
+            locked_req.pin: tuple(locked_req.iter_artifacts())
+            for locked_req in locked_resolve.locked_requirements
+        }
+        assert set(pins) == set(locked_requirements)
+        assert all(
+            isinstance(artifact, FileArtifact) and artifact.is_source
+            for artifact in itertools.chain.from_iterable(locked_requirements.values())
+        )
+
+    run_pex3(
+        "lock",
+        "create",
+        "--no-wheel",
+        "--constraints",
+        constraints,
+        "ansicolors",
+        "cowsay",
+        "--indent",
+        "2",
+        "-o",
+        lock,
+    ).assert_success()
+    assert_lock(
+        Pin(ProjectName("ansicolors"), Version("1.1.7")), Pin(ProjectName("cowsay"), Version("5.0"))
+    )
+
+    # N.B.: Pre-fix this test would lead to an artifact comparison assertion for cowsay, which is
+    # expected to be unmodified by the lock update.
+    #
+    # E       Traceback (most recent call last):
+    # E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/result.py", line 105, in catch
+    # E           return func(*args, **kwargs)
+    # E                  ^^^^^^^^^^^^^^^^^^^^^
+    # E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/resolve/lockfile/updater.py", line 320, in update_resolve
+    # E           assert updated_requirement.artifact == locked_requirement.artifact
+    # E                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    # E       AssertionError
+    # E       Encountered 1 error updating /tmp/pytest-of-jsirois/pytest-8/test_update_sdists_not_updated0/lock.json:
+    # E       1.) cp311-cp311-manylinux_2_35_x86_64:
+    run_pex3("lock", "update", "-v", "-p", "ansicolors<1.1.9", lock).assert_success()
+    assert_lock(
+        Pin(ProjectName("ansicolors"), Version("1.1.8")), Pin(ProjectName("cowsay"), Version("5.0"))
+    )