Repository: pex-tool/pex

Issue:
Strange issue with et-xmlfile dependency
Hello, since pex > 2.1.24, I'm facing issues with a dependencies of the `tablib` package being an old package et-xmlfile

Here is the log when I run the PEX file 
```± python dist/dcma
Traceback (most recent call last):
  File "/Users/b.vallet/Workspace/python-projects/dcma/dist/dcma/.bootstrap/pex/pex.py", line 476, in execute
  File "/Users/b.vallet/Workspace/python-projects/dcma/dist/dcma/.bootstrap/pex/pex.py", line 132, in activate
  File "/Users/b.vallet/Workspace/python-projects/dcma/dist/dcma/.bootstrap/pex/pex.py", line 119, in _activate
  File "/Users/b.vallet/Workspace/python-projects/dcma/dist/dcma/.bootstrap/pex/environment.py", line 378, in activate
  File "/Users/b.vallet/Workspace/python-projects/dcma/dist/dcma/.bootstrap/pex/environment.py", line 702, in _activate
  File "/Users/b.vallet/Workspace/python-projects/dcma/dist/dcma/.bootstrap/pex/environment.py", line 612, in resolve
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /Users/b.vallet/Workspace/python-projects/dcma/dist/dcma.
Needed macosx_10_16_x86_64-cp-36-cp36m compatible dependencies for:
 1: python>=2.6.0
    Required by:
      et-xmlfile 1.0.1
    But this pex had no 'python' distributions.
```

My pex is build with a 'standard' way via tox:


```[testenv:bundle]
deps = pex
setenv =
    LANG=en_US.UTF-8
commands =
    # Collect the statics to be embedded in the sdist and PEX file (via the MANIFEST)
    {envpython} manage.py collectstatic --noinput --clear
    # Creates a source archive in sdist/
    {envpython} setup.py sdist --dist-dir=sdist --format=gztar
    # Build exec file and save it in dist/
    {envpython} setup.py bdist_pex --bdist-dir=dist --pex-args='--disable-cache --index-url=[custom criteo pypi mirror]' --bdist-all
```

Python version is 3.6.8

When building it with pex==2.1.24, I have no issue.

Do you have any clue if I can fix it in my package or the issue comes from the dependency ?

Thanks a lot


Discussion:
Hrm, that package has a bad `setup.py`:
```python
#!/usr/bin/env python

"""Setup script for packaging et_xmfile.

Requires setuptools.

To build the setuptools egg use
    python setup.py bdist_egg
and either upload it to the PyPI with:
    python setup.py upload
or upload to your own server and register the release with PyPI:
    python setup.py register

A source distribution (.zip) can be built with
    python setup.py sdist --format=zip

That uses the manifest.in file for data files rather than searching for
them here.

"""
import codecs
import sys
import os
import warnings
if sys.version_info < (2, 6):
    raise Exception("Python >= 2.6 is required.")
elif sys.version_info[:2] == (3, 2):
    warnings.warn("Python 3.2 is no longer officially supported")

from setuptools import setup, Extension, find_packages
import re

here = os.path.abspath(os.path.dirname(__file__))
try:
    with codecs.open(os.path.join(here, 'README.rst'), encoding="utf-8") as f:
        README = f.read()
except IOError:
    README = ''

from et_xmlfile import (
    __author__,
    __license__,
    __author_email__,
    __url__,
    __version__
)


setup(name='et_xmlfile',
    packages=find_packages(),
    # metadata
    version=__version__,
    description="An implementation of lxml.xmlfile for the standard library",
    long_description=README,
    author=__author__,
    author_email=__author_email__,
    url=__url__,
    license=__license__,
    requires=[
        'python (>=2.6.0)',
        ],
    classifiers=[
                 'Development Status :: 5 - Production/Stable',
                 'Operating System :: MacOS :: MacOS X',
                 'Operating System :: Microsoft :: Windows',
                 'Operating System :: POSIX',
                 'License :: OSI Approved :: MIT License',
                 'Programming Language :: Python',
                 'Programming Language :: Python :: 2.6',
                 'Programming Language :: Python :: 2.7',
                 'Programming Language :: Python :: 3.3',
                 'Programming Language :: Python :: 3.4',
                 ],
    )
```

Note the `requires=['python (>=2.6.0)']`. The `requires` kwarg is old, see here: https://setuptools.readthedocs.io/en/latest/references/keywords.html
```
requires

    Warning

    requires is superseded by install_requires and should not be used anymore.

```

The fact that `requires` is old is not the problem, its that they use that field to restrict the Python version and not for 3rdparty requirements as its intended for. They should be using `python_requires` - in particular `python_requires='>=2.6,!3.2'`.

I'm not really sure what Pex can do here - they're simply using the setup.py metadata incorrectly. It just happens that Pex never used to read this metadata.

You can only use the PEX_IGNORE_ERRORS=1 backdoor at runtime like so:
```
$ pex --version
2.1.28
^jsirois@gill ~/dev/pantsbuild/pex (master) $ pex et-xmlfile==1.0.1 -oet-xmlfile.pex
^jsirois@gill ~/dev/pantsbuild/pex (master) $ ./et-xmlfile.pex 
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/pex/et-xmlfile.pex/.bootstrap/pex/pex.py", line 476, in execute
  File "/home/jsirois/dev/pantsbuild/pex/et-xmlfile.pex/.bootstrap/pex/pex.py", line 132, in activate
  File "/home/jsirois/dev/pantsbuild/pex/et-xmlfile.pex/.bootstrap/pex/pex.py", line 119, in _activate
  File "/home/jsirois/dev/pantsbuild/pex/et-xmlfile.pex/.bootstrap/pex/environment.py", line 378, in activate
  File "/home/jsirois/dev/pantsbuild/pex/et-xmlfile.pex/.bootstrap/pex/environment.py", line 702, in _activate
  File "/home/jsirois/dev/pantsbuild/pex/et-xmlfile.pex/.bootstrap/pex/environment.py", line 609, in resolve
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/jsirois/dev/pantsbuild/pex/et-xmlfile.pex.
Needed manylinux_2_32_x86_64-cp-39-cp39 compatible dependencies for:
 1: python>=2.6.0
    Required by:
      et-xmlfile 1.0.1
    But this pex had no 'python' distributions.
^jsirois@gill ~/dev/pantsbuild/pex (master) $ PEX_IGNORE_ERRORS=1 ./et-xmlfile.pex 
Python 3.9.1 (default, Dec 13 2020, 11:55:53) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import et_xmlfile
>>> et_xmlfile.__file__
'/home/jsirois/.pex/installed_wheels/038b58304533e7853e0ae3dc5c617a09f7d1342c/et_xmlfile-1.0.1-py3-none-any.whl/et_xmlfile/__init__.py'
>>> 
now exiting InteractiveConsole...
```

Or else at build-time using by adding `--ignore-errors` to your `--pex-args`.
@Djailla if either the build-time `--ignore-errors` or the runtime `PEX_IGNORE_ERRORS=1` env var solutions work for you, let me know and I'll mark this issue as an answered question.
@Djailla I'm going ahead and closing this as an answered question. Please speak up if that doesn't work for you.
I'm re-opening this since we've had another report of needing `et_xmlfile` (indirectly), but being unable to use it with modern Pex.

I've done some research in the Pip and setuptools codebase that shows this metadata is acknowledged and even transcribed, but is otherwise ignored. An experiment confirmed this - changing `install_requires` to `requires` in a `setup.py` configuration causes the resulting built distribution, when installed by pip, to not have its dependencies also installed - they are simply skipped!

Afaict from the standards - this is buggy behavior on the part of pip / setuptools:
+ [1.0](https://www.python.org/dev/peps/pep-0241/#fields): No requirement metadata specified yet.
+ [1.1](https://www.python.org/dev/peps/pep-0314/#summary-of-differences-from-pep-241): `Requires` field introduced.
+ [1.2](https://www.python.org/dev/peps/pep-0345/#summary-of-differences-from-pep-314): `Requires-Dist` field introduced and `Requires` field deprecated.
+ [2.1](https://www.python.org/dev/peps/pep-0566/#summary-of-differences-from-pep-345) Says that fields are now specified by the core metadata specification: https://packaging.python.org/specifications/core-metadata/ - that implies `Requires` is no longer valid.

For `et_xmlfile` 1.0.1 which is distributed as an sdist on PyPI we have:
```
$ grep -E "Metadata-Version|Name|Version|Requires" PKG-INFO
Metadata-Version: 1.1
Name: et_xmlfile
Version: 1.0.1
Requires: python (>=2.6.0)
```
So it uses 1.1 metadata wherein `Requires` is valid metadata. Pip and setuptools should interpret it as such and emit more modern metadata if that is their wont, converting Requires to Requires-Dist which 1.2 says is the replacement. 

Since Pex is dealing with the _results_ of `pip {download,wheel, install}` it really can only just go along with this strategy of incorrectly considering `Metadata-Version`. This just means Pex fails to create correct Pexes for the same requirements Pip fails to perform correct resolves for. Without also supporting this same bug during Pexes own validation, it just makes the situation worse for any distribtuon - like `et_xmlfile` that has no real requirements - however you spell them - and so happens not to be broken by Pip / setuptols broken handling (ignoring) of its `Requires` metadata.

Resolving patch:
diff --git a/pex/dist_metadata.py b/pex/dist_metadata.py
--- a/pex/dist_metadata.py
+++ b/pex/dist_metadata.py
@@ -11,9 +11,10 @@
 from email.message import Message
 from email.parser import Parser
 from io import StringIO
+from textwrap import dedent
 
 from pex import pex_warnings
-from pex.common import open_zip
+from pex.common import open_zip, pluralize
 from pex.compatibility import to_unicode
 from pex.third_party.packaging.specifiers import SpecifierSet
 from pex.third_party.pkg_resources import DistInfoDistribution, Distribution, Requirement
@@ -21,7 +22,7 @@
 
 if TYPE_CHECKING:
     import attr  # vendor:skip
-    from typing import Dict, Iterator, Optional, Union
+    from typing import Dict, Iterator, Optional, Union, List
 
     DistributionLike = Union[Distribution, str]
 else:
@@ -232,15 +233,15 @@ def requires_python(dist):
     return SpecifierSet(python_requirement)
 
 
-def requires_dists(
-    dist,  # type: DistributionLike
-    include_1_1_requires=True,  # type: bool
-):
-    # type: (...) -> Iterator[Requirement]
+def requires_dists(dist):
+    # type: (DistributionLike) -> Iterator[Requirement]
     """Examines dist for and returns any declared requirements.
 
-    Looks for `Requires-Dist` metadata and, optionally, the older `Requires` metadata if
-    `include_1_1_requires`.
+    Looks for `Requires-Dist` metadata.
+
+    The older `Requires` metadata is intentionally ignored, athough we do log a warning if it is
+    found to draw attention to this ~work-around and the associated issue in case any new data
+    comes in.
 
     See:
     + https://www.python.org/dev/peps/pep-0345/#requires-dist-multiple-use
@@ -256,6 +257,28 @@ def requires_dists(
     for requires_dist in pkg_info.get_all("Requires-Dist", ()):
         yield Requirement.parse(requires_dist)
 
-    if include_1_1_requires:
-        for requires in pkg_info.get_all("Requires", ()):
-            yield Requirement.parse(requires)
+    legacy_requires = pkg_info.get_all("Requires", [])  # type: List[str]
+    if legacy_requires:
+        name_and_version = project_name_and_version(dist)
+        project_name = name_and_version.project_name if name_and_version else dist
+        pex_warnings.warn(
+            dedent(
+                """\
+                Ignoring {count} `Requires` {field} in {dist} metadata:
+                {requires}
+
+                You may have issues using the '{project_name}' distribution as a result.
+                More information on this workaround can be found here:
+                  https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585
+                """
+            ).format(
+                dist=dist,
+                project_name=project_name,
+                count=len(legacy_requires),
+                field=pluralize(legacy_requires, "field"),
+                requires=os.linesep.join(
+                    "{index}.) Requires: {req}".format(index=index, req=req)
+                    for index, req in enumerate(legacy_requires, start=1)
+                ),
+            )
+        )


Test patch:
diff --git a/tests/test_dist_metadata.py b/tests/test_dist_metadata.py
--- a/tests/test_dist_metadata.py
+++ b/tests/test_dist_metadata.py
@@ -7,6 +7,7 @@
 import tarfile
 import warnings
 from contextlib import contextmanager
+from textwrap import dedent
 
 import pytest
 
@@ -18,6 +19,7 @@
     requires_dists,
     requires_python,
 )
+from pex.orderedset import OrderedSet
 from pex.pex_warnings import PEXWarning
 from pex.pip import get_pip
 from pex.third_party.packaging.specifiers import SpecifierSet
@@ -250,3 +252,26 @@ def test_requires_dists_none(pygoogleearth_zip_sdist):
     with example_distribution("MarkupSafe-1.0-cp27-cp27mu-linux_x86_64.whl") as (wheel_path, dist):
         assert [] == list(requires_dists(wheel_path))
         assert [] == list(requires_dists(dist))
+
+    # This tests a strange case detailed here:
+    #   https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585
+    with downloaded_sdist("et-xmlfile==1.0.1") as sdist, warnings.catch_warnings(
+        record=True
+    ) as events:
+        assert [] == list(requires_dists(sdist))
+        assert len(events) == 1
+        warning = events[0]
+        assert PEXWarning == warning.category
+        assert (
+            dedent(
+                """\
+                Ignoring 1 `Requires` field in {sdist} metadata:
+                1.) Requires: python (>=2.6.0)
+
+                You may have issues using the 'et_xmlfile' distribution as a result.
+                More information on this workaround can be found here:
+                  https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585
+                """
+            ).format(sdist=sdist)
+            == str(warning.message)
+        )
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -3186,3 +3186,11 @@ def vendored_toplevel(isolated_dir):
     assert (
         ansicolors_pex_isolated_vendoreds == tally_isolated_vendoreds()
     ), "Expecting no new Pex isolations."
+
+
+def test_requires_metadata_issues_1201(tmpdir):
+    # type: (Any) -> None
+    pex_file = os.path.join(str(tmpdir), "et-xmlfile.pex")
+    result = run_pex_command(args=["et-xmlfile==1.0.1", "-o", pex_file])
+    result.assert_success()
+    subprocess.check_call(args=[pex_file, "-c", "import et_xmlfile"])