Repository: pantsbuild/pants

Issue:
`pex.resolve.lockfile.ParseError` while building pex from lockfile
**Describe the bug**
I have the following in `requirements.txt`:
```
gem-metrics @ git+https://github.com/GEM-benchmark/GEM-metrics.git@1d6386ae7263b4542214142204de4d1f37a4b526
```
That results in the following snippet generated in the lockfile:
```
          "project_name": "gem-metrics",
          "requires_dists": [
            "bert-score; extra == \"heavy\"",
            "bleurt@ git+https://github.com/google-research/bleurt.git; extra == \"heavy\"",
            "datasets>=1.2.1",
            "diskcache>=5.2.1",
            "logzero",
            "nltk",
            "nubia-score; extra == \"heavy\"",
            "numpy",
            "pycountry",
            "questeval==0.2.4; extra == \"heavy\"",
            "rouge-score",
            "sacrebleu>=2.0.0",
            "sacremoses"
          ],
```

However, an attempt to use this lockfile fails with this exception:
```
Traceback (most recent call last):
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/requirements.py", line 106, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3205, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pex.vendor._vendored.setuptools.pkg_resources._vendor.pyparsing.ParseException: Expected stringEnd (at char 59), (line:1, col:60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3133, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/requirements.py", line 110, in __init__
    requirement_string[e.loc:e.loc + 8]))
pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'extra =='"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/resolve/lockfile/json_codec.py", line 116, in parse_requirement
    return Requirement.parse(path_mappings.maybe_reify(raw_requirement))
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3180, in parse
    req, = parse_requirements(s)
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3126, in parse_requirements
    yield Requirement(line)
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3135, in __init__
    raise RequirementParseError(str(e))
pex.vendor._vendored.setuptools.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'extra =='"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/richard/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 504, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/richard/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 409, in _wrap_coverage
    return runner(*args)
  File "/Users/richard/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 440, in _wrap_profiling
    return runner(*args)
  File "/Users/richard/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 560, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/richard/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 696, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/Users/richard/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 728, in execute_pkg_resources
    return runner()
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 766, in main
    env=env,
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 786, in do_main
    cache=ENV.PEX_ROOT,
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 622, in build_pex
    lock = try_(resolver_configuration.parse_lock())
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/resolve/resolver_options.py", line 392, in <lambda>
    parse_lock=lambda: parse_lockfile(options),
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/resolve/resolver_options.py", line 503, in parse_lockfile
    return lockfile.load(path, path_mappings=path_mappings)
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/resolve/lockfile/__init__.py", line 76, in load
    return json_codec.load(lockfile_path=lockfile_path, path_mappings=path_mappings)
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/resolve/lockfile/json_codec.py", line 260, in load
    return loads(fp.read(), source=lockfile_path, path_mappings=path_mappings)
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/resolve/lockfile/json_codec.py", line 222, in loads
    get("requires_dists", list, data=req, path=req_path)
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/resolve/lockfile/json_codec.py", line 221, in <listcomp>
    for i, requires_dist in enumerate(
  File "/Users/richard/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/resolve/lockfile/json_codec.py", line 119, in parse_requirement
    "The requirement string at '{path}' is invalid: {err}".format(path=path, err=e)
pex.resolve.lockfile.ParseError: The requirement string at '.locked_resolves[0][16]["requires_dists"][1]' is invalid: Invalid requirement, parse error at "'extra =='"
```


**Pants version**
2.11.0

**OS**
MacOS

**Additional info**
https://gist.github.com/rshin/160bfb09b597e127b7583154b9a1e16b contains all files to reproduce this bug.


Discussion:
It looks like this is inadvertently fixed in Pex 2.1.89 as a result of it moving away from any use of `pkg_resources`. Apparently `pkg_resource.Requirement` has an invalid `__str__` for this vcs + extras case whereas `packaging.requirements.Requirement` is OK:

Repro at Pex 2.1.84:
```
$ pex pex==2.1.84 -cpex3 -- lock create --interpreter-constraint ">=3.7,<4" --style universal --resolver-version pip-2020-resolver "gem-metrics @ git+https://github.com/GEM-benchmark/GEM-metrics.git@1d6386ae7263b4542214142204de4d1f37a4b526" -o lock.2.1.84.json --indent 2
$ pex pex==2.1.84 -cpex -- --python python3.7 --lock lock.2.1.84.json -o gem-metrics.pex
...
pex.resolve.lockfile.ParseError: The requirement string at '.locked_resolves[0][16]["requires_dists"][1]' is invalid: Invalid requirement, parse error at "'extra =='"
```
And same in 2.1.88:
```
$ pex pex==2.1.88 -cpex3 -- lock create --interpreter-constraint ">=3.7,<4" --style universal --resolver-version pip-2020-resolver "gem-metrics @ git+https://github.com/GEM-benchmark/GEM-metrics.git@1d6386ae7263b4542214142204de4d1f37a4b526" -o lock.2.1.88.json --indent 2
$ pex pex==2.1.88 -cpex -- --python python3.7 --lock lock.2.1.88.json -o gem-metrics.pex
...
pex.resolve.lockfile.ParseError: The requirement string at '.locked_resolves[0][16]["requires_dists"][1]' is invalid: Invalid requirement, parse error at "'extra =='"
```
But fixed in 2.1.89:
```
$ pex pex==2.1.89 -cpex3 -- lock create --interpreter-constraint ">=3.7,<4" --style universal --resolver-version pip-2020-resolver "gem-metrics @ git+https://github.com/GEM-benchmark/GEM-metrics.git@1d6386ae7263b4542214142204de4d1f37a4b526" -o lock.2.1.89.json --indent 2
$ pex pex==2.1.89 -cpex -- --python python3.7 --lock lock.2.1.89.json -o gem-metrics.pex
$ ./gem-metrics.pex -c 'import gem_metrics; print(gem_metrics.__file__)'
/home/jsirois/.pex/installed_wheels/653661a1b177c80bcbb6f5a9d62706ec87d68a5de3dd50f395bca987f5437670/gem_metrics-0.1.dev0-py3-none-any.whl/gem_metrics/__init__.py
```

The difference:
```diff3
$ diff3 lock.2.1.84.json lock.2.1.88.json lock.2.1.89.json 
====3
1:1115c
2:1115c
              "bleurt@ git+https://github.com/google-research/bleurt.git; extra == \"heavy\"",
3:1115c
              "bleurt@ git+https://github.com/google-research/bleurt.git ; extra == \"heavy\"",
====
1:3586c
    "pex_version": "2.1.84",
2:3586c
    "pex_version": "2.1.88",
3:3586c
    "pex_version": "2.1.89",
```


Yeah, a hand-edit of the 2.1.84 lockfile nets the same success:
```
$ cp lock.2.1.84.json lock.2.1.84-edited.json
$ vi lock.2.1.84-edited.json 
$ diff lock.2.1.84.json lock.2.1.84-edited.json 
1115c1115
<             "bleurt@ git+https://github.com/google-research/bleurt.git; extra == \"heavy\"",
---
>             "bleurt@ git+https://github.com/google-research/bleurt.git ; extra == \"heavy\"",
3598c3598
< }
\ No newline at end of file
---
> }
$ pex pex==2.1.84 -cpex -- --python python3.7 --lock lock.2.1.84-edited.json -o gem-metrics.pex
$ ./gem-metrics.pex -c 'import gem_metrics; print(gem_metrics.__file__)'
/home/jsirois/.pex/installed_wheels/653661a1b177c80bcbb6f5a9d62706ec87d68a5de3dd50f395bca987f5437670/gem_metrics-0.1.dev0-py3-none-any.whl/gem_metrics/__init__.py
```
Thanks for looking into this! As a short-term workaround, is there a way to opt into using Pex 2.1.89 with Pants 2.11.0? I see from https://github.com/pantsbuild/pants/blob/main/src/python/pants/notes/2.11.x.md that Pants 2.11.1 should be switching to Pex 2.1.88, but you showed above that 2.1.88 still has the same bug.
There is :) See https://www.pantsbuild.org/docs/restricted-internet-access#binary-tools for how to set the options at https://www.pantsbuild.org/docs/reference-pex-cli. The downloaded file comes from https://github.com/pantsbuild/pex/releases/tag/v2.1.89
I'll be upgrading Pants to [Pex 2.1.90](https://github.com/pantsbuild/pex/issues/1769) later today which should net 2.11 & 2.12 rcs with the fix in addition to the 2.13 dev branch. In the meantime, speak up if you stumble following Eric's advice to work-around.

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -39,9 +39,9 @@ class PexCli(TemplatedExternalTool):
     name = "pex"
     help = "The PEX (Python EXecutable) tool (https://github.com/pantsbuild/pex)."
 
-    default_version = "v2.1.88"
+    default_version = "v2.1.90"
     default_url_template = "https://github.com/pantsbuild/pex/releases/download/{version}/pex"
-    version_constraints = ">=2.1.88,<3.0"
+    version_constraints = ">=2.1.90,<3.0"
 
     @classproperty
     def default_known_versions(cls):
@@ -50,8 +50,8 @@ def default_known_versions(cls):
                 (
                     cls.default_version,
                     plat,
-                    "ea8f696dac3b1ddac2fe7af288251c63ca4ea0dd16808f01e60b26b21c84a260",
-                    "3748292",
+                    "2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66",
+                    "3755345",
                 )
             )
             for plat in ["macos_arm64", "macos_x86_64", "linux_x86_64"]


Test patch: