Repository: pypa/pip

Issue:
Pip wheel doesn't properly do ABI tag for py2.7
It appears that using a debug build of python 2.7 doesn't mark the wheels built using it in any special way. 

Pip would install them on a regular python 2.7 (if they would be on an package index) and then later on imports for C extensions would fail.

ref @dstufft wheel caching



Discussion:
This is up to bdist_wheel to do, not pip IMO.

(unless we don't _honour_ markers for debug python). We need to honour it, of course.

The SOABI thing is new in Python 3 which the wheel spec uses (and the wheel tool and pip) which works fine, however there is no SOABI api in Python 2 so nothing currently creates or looks for ABI specific wheels (instead we look for CPython X.Y specific wheels). The outcome of this ticket would be to find a way to emulate the SOABI api on Python 2.x so we can start looking for ABI tagged wheels on Python 2.x. It would also involve coordination with wheel to add the same emulation there and get it to generate wheels tagged with an ABI.

Ok. So _should_ this be a release blocker? Its no worse than the behaviour with pip wheel.

It's no worse than `pip wheel`, except using `pip wheel` is opt in while using the new auto wheels is opt out. Whether it's a release blocker or not would, in my opinion, depend on how likely it is that you're going to have two Pythons of the same minor version with different ABIs installed on the same machine.

In the IRC discussion that lead to this, I believe it was said (at least that's how I understood it) that it's not uncommon to run two different ABI Python when it comes to debug builds so I marked it as a release-blocker. If my understanding was wrong we can drop the release-block part of this.

Of course the above is really, two minor versions, _in the 2.x line_, installed on the same machine. It should already work fine on 3.x.

## ok so - python2.7-dbg on ubuntu says this:

Description-en: Debug Build of the Python Interpreter (version 2.7)
 The package holds two things:
 .
- A Python interpreter configured with --pydebug. Dynamically loaded modules
  are searched as <foo>_d.so first. Third party extensions need a separate
  build to be used by this interpreter.
- Debug information for standard python interpreter and extensions.

---

So having that installed doesn't magically get folk _using_ the debug build.

My argument is that anyone using the debug python2 build knows what they're doing. They might not know that the C extensions can't be automatically discriminated between, but they won't accidentally switch from python to python-dbg.

As well as debug builds, there's also 32-bit and 64-bit Pythons on the same Windows machine. I know this can be done (for example, Appveyor includes this) but I don't know if it would ever be a problem in practice (Appveyor shouldn't be an issue because it runs tests in isolated VMs).

On Fri, Apr 17, 2015 at 4:17 AM, rbtcollins notifications@github.com
wrote:

> My argument is that anyone using the debug python2 build knows what
> they're doing. They might not know that the C extensions can't be
> automatically discriminated between, but they won't accidentally switch
> from python to python-dbg.

​The element of surprise is still there. You should have seen my face when
I finally realized the import issue was caused by me building wheels with
debug python couple days before.​

Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

Sure. I'm just at a fairly big loss about what to do here.

My assumptions:
- most (call it 99%) of devs won't install the debug version of Python.
- the benefit to the ones that don't is unambiguous
- the ones that do are into a grey area that CPython doesn't really help with, and is documented as having dragons.

I might be the only person who understands wheel tagging in bdist_wheel. If that doesn't change then you are going to have to wait for me to implement the debug / unicode detection so that bdist_wheel can generate a better Python ABI tag for Python 2.7. Otherwise someone else should figure it out and submit a pull request to bdist_wheel.

So - my concern here is that this is tagged release-blocker. I don't think it should be.


Resolving patch:
diff --git a/pip/pep425tags.py b/pip/pep425tags.py
--- a/pip/pep425tags.py
+++ b/pip/pep425tags.py
@@ -15,6 +15,14 @@
 _osx_arch_pat = re.compile(r'(.+)_(\d+)_(\d+)_(.+)')
 
 
+def get_config_var(var):
+    try:
+        return sysconfig.get_config_var(var)
+    except IOError as e:  # Issue #1074
+        warnings.warn("{0}".format(e), RuntimeWarning)
+        return None
+
+
 def get_abbr_impl():
     """Return abbreviated implementation name."""
     if hasattr(sys, 'pypy_version_info'):
@@ -30,7 +38,67 @@ def get_abbr_impl():
 
 def get_impl_ver():
     """Return implementation version."""
-    return ''.join(map(str, sys.version_info[:2]))
+    impl_ver = get_config_var("py_version_nodot")
+    if not impl_ver or get_abbr_impl() == 'pp':
+        impl_ver = ''.join(map(str, get_impl_version_info()))
+    return impl_ver
+
+
+def get_impl_version_info():
+    """Return sys.version_info-like tuple for use in decrementing the minor
+    version."""
+    if get_abbr_impl() == 'pp':
+        # as per https://github.com/pypa/pip/issues/2882
+        return (sys.version_info[0], sys.pypy_version_info.major,
+                sys.pypy_version_info.minor)
+    else:
+        return sys.version_info[0], sys.version_info[1]
+
+
+def get_flag(var, fallback, expected=True, warn=True):
+    """Use a fallback method for determining SOABI flags if the needed config
+    var is unset or unavailable."""
+    val = get_config_var(var)
+    if val is None:
+        if warn:
+            warnings.warn("Config variable '{0}' is unset, Python ABI tag may "
+                          "be incorrect".format(var), RuntimeWarning, 2)
+        return fallback()
+    return val == expected
+
+
+def get_abi_tag():
+    """Return the ABI tag based on SOABI (if available) or emulate SOABI
+    (CPython 2, PyPy)."""
+    soabi = get_config_var('SOABI')
+    impl = get_abbr_impl()
+    if not soabi and impl in ('cp', 'pp') and hasattr(sys, 'maxunicode'):
+        d = ''
+        m = ''
+        u = ''
+        if get_flag('Py_DEBUG',
+                    lambda: hasattr(sys, 'gettotalrefcount'),
+                    warn=(impl == 'cp')):
+            d = 'd'
+        if get_flag('WITH_PYMALLOC',
+                    lambda: impl == 'cp',
+                    warn=(impl == 'cp')):
+            m = 'm'
+        if get_flag('Py_UNICODE_SIZE',
+                    lambda: sys.maxunicode == 0x10ffff,
+                    expected=4,
+                    warn=(impl == 'cp' and
+                          sys.version_info < (3, 3))) \
+                and sys.version_info < (3, 3):
+            u = 'u'
+        abi = '%s%s%s%s%s' % (impl, get_impl_ver(), d, m, u)
+    elif soabi and soabi.startswith('cpython-'):
+        abi = 'cp' + soabi.split('-')[1]
+    elif soabi:
+        abi = soabi.replace('.', '_').replace('-', '_')
+    else:
+        abi = None
+    return abi
 
 
 def get_platform():
@@ -51,23 +119,19 @@ def get_supported(versions=None, noarch=False):
     # Versions must be given with respect to the preference
     if versions is None:
         versions = []
-        major = sys.version_info[0]
+        version_info = get_impl_version_info()
+        major = version_info[:-1]
         # Support all previous minor Python versions.
-        for minor in range(sys.version_info[1], -1, -1):
-            versions.append(''.join(map(str, (major, minor))))
+        for minor in range(version_info[-1], -1, -1):
+            versions.append(''.join(map(str, major + (minor,))))
 
     impl = get_abbr_impl()
 
     abis = []
 
-    try:
-        soabi = sysconfig.get_config_var('SOABI')
-    except IOError as e:  # Issue #1074
-        warnings.warn("{0}".format(e), RuntimeWarning)
-        soabi = None
-
-    if soabi and soabi.startswith('cpython-'):
-        abis[0:0] = ['cp' + soabi.split('-')[1]]
+    abi = get_abi_tag()
+    if abi:
+        abis[0:0] = [abi]
 
     abi3s = set()
     import imp


Test patch:
diff --git a/tests/unit/test_wheel.py b/tests/unit/test_wheel.py
--- a/tests/unit/test_wheel.py
+++ b/tests/unit/test_wheel.py
@@ -1,5 +1,6 @@
 """Tests for wheel binary packages and .dist-info."""
 import os
+import sys
 
 import pytest
 from mock import patch, Mock
@@ -294,6 +295,52 @@ def test_version_underscore_conversion(self):
 
 class TestPEP425Tags(object):
 
+    def mock_get_config_var(self, **kwd):
+        """
+        Patch sysconfig.get_config_var for arbitrary keys.
+        """
+        import pip.pep425tags
+
+        get_config_var = pip.pep425tags.sysconfig.get_config_var
+
+        def _mock_get_config_var(var):
+            if var in kwd:
+                return kwd[var]
+            return get_config_var(var)
+        return _mock_get_config_var
+
+    def abi_tag_unicode(self, flags, config_vars):
+        """
+        Used to test ABI tags, verify correct use of the `u` flag
+        """
+        import pip.pep425tags
+
+        config_vars.update({'SOABI': None})
+        base = pip.pep425tags.get_abbr_impl() + pip.pep425tags.get_impl_ver()
+
+        if sys.version_info < (3, 3):
+            config_vars.update({'Py_UNICODE_SIZE': 2})
+            mock_gcf = self.mock_get_config_var(**config_vars)
+            with patch('pip.pep425tags.sysconfig.get_config_var', mock_gcf):
+                abi_tag = pip.pep425tags.get_abi_tag()
+                assert abi_tag == base + flags
+
+            config_vars.update({'Py_UNICODE_SIZE': 4})
+            mock_gcf = self.mock_get_config_var(**config_vars)
+            with patch('pip.pep425tags.sysconfig.get_config_var', mock_gcf):
+                abi_tag = pip.pep425tags.get_abi_tag()
+                assert abi_tag == base + flags + 'u'
+
+        else:
+            # On Python >= 3.3, UCS-4 is essentially permanently enabled, and
+            # Py_UNICODE_SIZE is None. SOABI on these builds does not include
+            # the 'u' so manual SOABI detection should not do so either.
+            config_vars.update({'Py_UNICODE_SIZE': None})
+            mock_gcf = self.mock_get_config_var(**config_vars)
+            with patch('pip.pep425tags.sysconfig.get_config_var', mock_gcf):
+                abi_tag = pip.pep425tags.get_abi_tag()
+                assert abi_tag == base + flags
+
     def test_broken_sysconfig(self):
         """
         Test that pep425tags still works when sysconfig is broken.
@@ -314,14 +361,9 @@ def test_no_hyphen_tag(self):
         """
         import pip.pep425tags
 
-        get_config_var = pip.pep425tags.sysconfig.get_config_var
-
-        def mock_soabi(var):
-            if var == 'SOABI':
-                return 'cpython-35m-darwin'
-            return get_config_var(var)
+        mock_gcf = self.mock_get_config_var(SOABI='cpython-35m-darwin')
 
-        with patch('pip.pep425tags.sysconfig.get_config_var', mock_soabi):
+        with patch('pip.pep425tags.sysconfig.get_config_var', mock_gcf):
             supported = pip.pep425tags.get_supported()
 
         for (py, abi, plat) in supported:
@@ -329,6 +371,30 @@ def mock_soabi(var):
             assert '-' not in abi
             assert '-' not in plat
 
+    def test_manual_abi_noflags(self):
+        """
+        Test that no flags are set on a non-PyDebug, non-Pymalloc ABI tag.
+        """
+        self.abi_tag_unicode('', {'Py_DEBUG': False, 'WITH_PYMALLOC': False})
+
+    def test_manual_abi_d_flag(self):
+        """
+        Test that the `d` flag is set on a PyDebug, non-Pymalloc ABI tag.
+        """
+        self.abi_tag_unicode('d', {'Py_DEBUG': True, 'WITH_PYMALLOC': False})
+
+    def test_manual_abi_m_flag(self):
+        """
+        Test that the `m` flag is set on a non-PyDebug, Pymalloc ABI tag.
+        """
+        self.abi_tag_unicode('m', {'Py_DEBUG': False, 'WITH_PYMALLOC': True})
+
+    def test_manual_abi_dm_flags(self):
+        """
+        Test that the `dm` flags are set on a PyDebug, Pymalloc ABI tag.
+        """
+        self.abi_tag_unicode('dm', {'Py_DEBUG': True, 'WITH_PYMALLOC': True})
+
 
 class TestMoveWheelFiles(object):
     """