Repository: pex-tool/pex

Issue:
Extend wheel ABI support for pypy.
pypy version:

```
$ pypy
Python 2.7.10 (246c9cf22037b11dc0e8c29ce3f291d3b8c5935a, Mar 10 2016, 18:28:19)
[PyPy 5.0.0 with GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>>
```

The following command fails:

```
$ pex --python=pypy psutil
**** Failed to install psutil-4.1.0. stdout:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.10-x86_64-2.7
creating build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/__init__.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_common.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_compat.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_psbsd.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_pslinux.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_psosx.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_psposix.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_pssunos.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_pswindows.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
creating build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/__init__.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/runner.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_bsd.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_linux.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_memory_leaks.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_misc.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_osx.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_posix.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_process.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_sunos.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_system.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_windows.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
running build_ext
building 'psutil._psutil_osx' extension
creating build/temp.macosx-10.10-x86_64-2.7
creating build/temp.macosx-10.10-x86_64-2.7/psutil
creating build/temp.macosx-10.10-x86_64-2.7/psutil/arch
creating build/temp.macosx-10.10-x86_64-2.7/psutil/arch/osx
cc -arch x86_64 -O2 -fPIC -Wimplicit -DPSUTIL_VERSION=410 -I/usr/local/Cellar/pypy/5.0.0/libexec/include -c psutil/_psutil_osx.c -o build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_osx.o
cc -arch x86_64 -O2 -fPIC -Wimplicit -DPSUTIL_VERSION=410 -I/usr/local/Cellar/pypy/5.0.0/libexec/include -c psutil/_psutil_common.c -o build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_common.o
cc -arch x86_64 -O2 -fPIC -Wimplicit -DPSUTIL_VERSION=410 -I/usr/local/Cellar/pypy/5.0.0/libexec/include -c psutil/arch/osx/process_info.c -o build/temp.macosx-10.10-x86_64-2.7/psutil/arch/osx/process_info.o
cc -shared -undefined dynamic_lookup -arch x86_64 build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_osx.o build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_common.o build/temp.macosx-10.10-x86_64-2.7/psutil/arch/osx/process_info.o -o build/lib.macosx-10.10-x86_64-2.7/psutil/_psutil_osx.pypy-41.so -framework CoreFoundation -framework IOKit
building 'psutil._psutil_posix' extension
cc -arch x86_64 -O2 -fPIC -Wimplicit -I/usr/local/Cellar/pypy/5.0.0/libexec/include -c psutil/_psutil_posix.c -o build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_posix.o
cc -shared -undefined dynamic_lookup -arch x86_64 build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.macosx-10.10-x86_64-2.7/psutil/_psutil_posix.pypy-41.so
installing to build/bdist.macosx-10.10-x86_64/wheel
running install
running install_lib
creating build/bdist.macosx-10.10-x86_64
creating build/bdist.macosx-10.10-x86_64/wheel
creating build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/__init__.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_common.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_compat.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psbsd.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_pslinux.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psosx.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psposix.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_pssunos.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psutil_osx.pypy-41.so -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psutil_posix.pypy-41.so -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_pswindows.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
creating build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/__init__.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/runner.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_bsd.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_linux.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_memory_leaks.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_misc.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_osx.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_posix.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_process.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_sunos.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_system.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_windows.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
running install_egg_info
running egg_info
writing psutil.egg-info/PKG-INFO
writing dependency_links to psutil.egg-info/dependency_links.txt
writing top-level names to psutil.egg-info/top_level.txt
reading manifest file 'psutil.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'psutil.egg-info/SOURCES.txt'
Copying psutil.egg-info to build/bdist.macosx-10.10-x86_64/wheel/psutil-4.1.0-py2.7.egg-info
running install_scripts

**** Failed to install psutil-4.1.0. stderr:
warning: no previously-included files matching '*' found under directory 'docs/_build'
Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "setup.py", line 284, in <module>
    main()
  File "setup.py", line 281, in main
    setup(**setup_args)
  File "/usr/local/Cellar/pypy/5.0.0/libexec/lib-python/2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/local/Cellar/pypy/5.0.0/libexec/lib-python/2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/pypy/5.0.0/libexec/lib-python/2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "build/bdist.macosx-10.10-x86_64/egg/wheel/bdist_wheel.py", line 213, in run
    archive_basename = self.get_archive_basename()
  File "build/bdist.macosx-10.10-x86_64/egg/wheel/bdist_wheel.py", line 161, in get_archive_basename
    impl_tag, abi_tag, plat_tag = self.get_tag()
  File "build/bdist.macosx-10.10-x86_64/egg/wheel/bdist_wheel.py", line 155, in get_tag
    assert tag == supported_tags[0]
AssertionError

Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 315, in execute
  File ".bootstrap/_pex/pex.py", line 250, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 282, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 358, in _execute
  File ".bootstrap/_pex/pex.py", line 416, in execute_entry
  File ".bootstrap/_pex/pex.py", line 434, in execute_pkg_resources
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/bin/pex.py", line 509, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/bin/pex.py", line 471, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/resolver.py", line 191, in resolve
    dist = self.build(package, resolvable.options)
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/resolver.py", line 248, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/resolver.py", line 160, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage(u'file:///Users/shimin/.pex/build/psutil-4.1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
```



Discussion:
pex + pypy + native deps has never been proven to be super reliable, but it seems to me that the error message you've pasted is actually coming from the translation phase where pex is mostly removed from the picture - e.g. the equivalent of:

```
wget $PSUTIL_URL
tar zxvf $PSUTIL_TARBALL
cd $PSUTIL_DIR
pypy setup.py bdist_wheel
```

if you manually run the setup.py in this fashion, do you hit the same AssertionError?

does it change if opt for eggs only (`--no-wheel`)?

for pex + pypy users, we also suggest using `--not-zip-safe` and `--always-write-cache` due to https://bitbucket.org/pypy/pypy/issues/1686/issue-with-our-zipimporter-module.

I will try these commands when I get in front of a computer, but just fyi,
`pip_pypy wheel psutil` works.
On Mar 16, 2016 3:45 PM, "Kris Wilson" notifications@github.com wrote:

> pex + pypy + native deps has never been proven to be super reliable, but
> it seems to me that the error message you've pasted is actually coming from
> the translation phase where pex is mostly removed from the picture - e.g.
> the equivalent of:
> 
> wget $PSUTIL_URL
> tar zxvf $PSUTIL_TARBALL
> cd $PSUTIL_DIR
> pypy setup.py bdist_wheel
> 
> if you manually run the setup.py in this fashion, do you hit the same
> AssertionError?
> 
> does it change if opt for eggs only (--no-wheel)?
> 
> for pex + pypy users, we also suggest using --not-zip-safe and
> --always-write-cache due to
> https://bitbucket.org/pypy/pypy/issues/1686/issue-with-our-zipimporter-module
> .
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
> https://github.com/pantsbuild/pex/issues/222#issuecomment-197590646

Running setup.py manually works.

`pex --python=pypy --no-wheel psutil` also works

It also doesn't work if I build the wheel first and then tell pex to use the local wheel.

```
shimin /tmp$ pip_pypy wheel -w deps psutil
Collecting psutil
  Using cached psutil-4.1.0.tar.gz
Building wheels for collected packages: psutil
  Running setup.py bdist_wheel for psutil ... done
  Stored in directory: /private/tmp/deps
Successfully built psutil
```

```
shimin /tmp$ ls deps
psutil-4.1.0-pp250-pypy_41-macosx_10_10_x86_64.whl
```

```
shimin /tmp$ pex --python pypy --disable-cache --no-pypi -f ./deps psutil -v -v -v
pex: crawling link i=0 link=Link('file:///private/tmp/deps') follow_links=False
Could not satisfy all requirements for psutil:
    psutil
```

It seems it's related to #213. Looking at this code: https://github.com/pantsbuild/pex/blob/0b1e5fdd1b6f33bd7ab1828c986c40e384dbb9e9/pex/pep425.py#L128, only 'none' is an accepted abi tag for pypy, but here we have `pypy_41`.

ah, makes sense. I think this is probably just a quick pypy-specific extension of #213 (which only covered cpython initially).

+1

this should now be vastly improved as of pex 1.4.0 (which adds impl/version/abi targeting), but I don't have a concrete repro to prove that.

you may try kicking the tires with the new version, with tag targeting usage described here: https://github.com/pantsbuild/pex/issues/281#issuecomment-389352166

I'm going to optimistically mark this as closed given much of the underlying resolver logic has changed, but feel free to reopen with a fresh repro if you're still having issues.

This still doesn't quite work, seems like something weird with the platform detection:

```
# In a virtualenv here:
$ python -V
Python 2.7.13 (6.0.0+dfsg-1, Apr 27 2018, 22:07:01)
[PyPy 6.0.0 with GCC 7.3.0]
$ pex --version
pex 1.4.0
$ pex -vvv -c twistd -o twisted.pex Twisted
…
pex: Validated Twisted-18.4.0.tar.bz2 (sha256=a4cc164a781859c74de47f17f0e85f4bce8a3321a9d0892c015c8f80c4158ad9)
pex: Target package WheelPackage('file:///tmp/tmpgwzUzE/Twisted-18.4.0-pp260-pypy_41-linux_x86_64.whl') is not compatible with [('pp260', 'pp260pypy_41', 'manylinux1_x86_64'), ('pp260', 'pp260pypy_41', 'linux_x86_64'), ('pp260', 'none', 'manylinux1_x86_64'), ('pp260', 'none', 'linux_x86_64'), ('py2', 'none', 'manylinux1_x86_64'), ('py2', 'none', 'linux_x86_64'), ('pp260', 'none', 'any'), ('pp2', 'none', 'any'), ('py260', 'none', 'any'), ('py2', 'none', 'any')]
Traceback (most recent call last):
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/bin/pex.py", line 694, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/bin/pex.py", line 624, in build_pex
    for dist in resolveds:
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 525, in resolve_multi
    use_manylinux=use_manylinux):
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 451, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 248, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 321, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 206, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
Untranslateable: Package SourcePackage(u'file:///home/mithrandi/.pex/build/Twisted-18.4.0.tar.bz2') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
```

Seems like `pp260` is landing up in `('pp260', 'pp260pypy_41', 'linux_x86_64')` twice for some reason.
Aha, the problem is this logic here:
https://github.com/pantsbuild/pex/blob/2208cb0418a4d37831fd688188b0a991efba4f63/pex/platforms.py#L50-L55

Since `pypy_41` does not begin with `pp260`, it gets prefixed with it, which is wrong for PyPy ABIs. If I pass `--platform linux_x86_64` explicitly then it works!

```
$ ./twisted.pex --version
twistd (the Twisted daemon) 18.4.0
Copyright (c) 2001-2018 Twisted Matrix Laboratories.
See LICENSE for details.
```

Resolving patch:
diff --git a/pex/platforms.py b/pex/platforms.py
--- a/pex/platforms.py
+++ b/pex/platforms.py
@@ -49,8 +49,10 @@ def create(cls, platform):
 
   @staticmethod
   def _maybe_prefix_abi(impl, version, abi):
-    # N.B. This permits users to pass in simpler extended platform strings like
-    # `linux-x86_64-cp-27-mu` vs e.g. `linux-x86_64-cp-27-cp27mu`.
+    if impl != 'cp':
+      return abi
+    # N.B. This permits CPython users to pass in simpler extended platform
+    # strings like `linux-x86_64-cp-27-mu` vs e.g. `linux-x86_64-cp-27-cp27mu`.
     impl_ver = ''.join((impl, version))
     return abi if abi.startswith(impl_ver) else ''.join((impl_ver, abi))
 


Test patch:
diff --git a/tests/test_platform.py b/tests/test_platform.py
--- a/tests/test_platform.py
+++ b/tests/test_platform.py
@@ -3,8 +3,6 @@
 
 import sys
 
-import pytest
-
 from pex.pep425tags import get_abi_tag, get_impl_tag
 from pex.platforms import Platform
 
@@ -57,7 +55,6 @@ def test_platform_supported_tags_manylinux():
   )
 
 
-@pytest.mark.skipif("(sys.version_info[0], sys.version_info[1]) == (2, 6)")
 def test_platform_supported_tags_osx_minimal():
   assert_tags(
     'macosx-10.4-x86_64',
@@ -84,3 +81,12 @@ def test_platform_supported_tags_osx_full():
       ('cp27', 'cp27m', 'macosx_10_12_x86_64'),
     ]
   )
+
+
+def test_pypy_abi_prefix():
+  assert_tags(
+    'linux-x86_64-pp-260-pypy_41',
+    [
+      ('pp260', 'pypy_41', 'linux_x86_64'),
+    ]
+  )