Repository: pantsbuild/pants

Issue:
Invalid abi tag in python_awslambda with python3.8
In python3.8 `m` suffix on abi tags can be omitted, e.g. psycopg2 is published as `psycopg2_binary-2.8.5-cp38-cp38-manylinux1_x86_64.whl` (abi tag is `cp38`, for python 3.7 it was `cp37m`).

When python_awslambda builds a pex, it adds the `m` suffix to the abi tag in the target platform. See https://github.com/pantsbuild/pants/blob/9960d8834a369f6dea1a1b940eb039d90843db37/src/python/pants/backend/awslambda/python/awslambda_python_rules.py#L71 .

I think adding the suffix only up until runtime version 3.7 should fix this (similar as it is done for the `u` suffix in python 2.7).
Alternatively a way to override the behaviour would be needed.


Discussion:
Interesting - I hadn't heard of this. @pyranja do you happen to know of any Python documentation that documents this change? I can't find it in PEP 425, the original definition of platform tags.

Is this causing a concrete issue for you? Specifically, are you feeling to resolve `psycopg2_binary-2.8.5-cp38-cp38-manylinux1_x86_64.whl` now?
@Eric-Arellano I found this issue https://bugs.python.org/issue36707 , where the change is discussed. To me it seems like this is a global change and should have an effect on all platform-dependent python packages.

I checked on my local python installations (ubuntu) using https://github.com/pypa/manylinux/blob/master/docker/build_scripts/python-tag-abi-tag.py

* python 3.8.2 -> `cp38-cp38`
* python 3.7.7 -> `cp37-cp37m`
* python 3.6.9 -> `cp36-cp36m`

I'm affected as the dependency resolution for `psycopg2-binary~=2.8` fails when using `python3.8` as `runtime` for `python_awslambda`:

```
ERROR: No matching distribution found for psycopg2-binary~=2.8
pid: 3822 -> /usr/bin/python2.7 /tmp/process-executionrOIJli/pex_root/pip.pex/49a647f5ed9e81dc8f85560600c53fbb4dfc60f0 --disable-pip-version-check --isolated --no-python-version-warning -q --cache-dir /tmp/process-executionrOIJli/pex_root download --dest /tmp/tmpOmUj6i/resolved_dists/manylinux2014_x86_64-cp-38-cp38m --index-url https://pypi.org/simple/ --platform manylinux2014_x86_64 --implementation cp --python-version 38 --abi cp38m --only-binary :all: boto3~=1.9 psycopg2-binary~=2.8 raised Executing /usr/bin/python2.7 /tmp/process-executionrOIJli/pex_root/pip.pex/49a647f5ed9e81dc8f85560600c53fbb4dfc60f0 --disable-pip-version-check --isolated --no-python-version-warning -q --cache-dir /tmp/process-executionrOIJli/pex_root download --dest /tmp/tmpOmUj6i/resolved_dists/manylinux2014_x86_64-cp-38-cp38m --index-url https://pypi.org/simple/ --platform manylinux2014_x86_64 --implementation cp --python-version 38 --abi cp38m --only-binary :all: boto3~=1.9 psycopg2-binary~=2.8 failed with 1
```

When executing the `pip3 download` with a fixed `--abi cp38`, resolution works:

```
pip3 --disable-pip-version-check --isolated --no-python-version-warning -q download --dest ./packages --index-url https://pypi.org/simple/ --platform manylinux2014_x86_64 --implementation cp --python-version 38 --abi cp38 --only-binary :all: psycopg2-binary~=2.8
```

Resolving patch:
diff --git a/src/python/pants/backend/awslambda/python/awslambda_python_rules.py b/src/python/pants/backend/awslambda/python/awslambda_python_rules.py
--- a/src/python/pants/backend/awslambda/python/awslambda_python_rules.py
+++ b/src/python/pants/backend/awslambda/python/awslambda_python_rules.py
@@ -68,7 +68,10 @@ async def create_python_awslambda(
     # (Running the "hello world" lambda in the example code will report the platform, and can be
     # used to verify correctness of these platform strings.)
     py_major, py_minor = field_set.runtime.to_interpreter_version()
-    platform = f"manylinux2014_x86_64-cp-{py_major}{py_minor}-cp{py_major}{py_minor}m"
+    platform = f"manylinux2014_x86_64-cp-{py_major}{py_minor}-cp{py_major}{py_minor}"
+    # set pymalloc ABI flag - this was removed in python 3.8 https://bugs.python.org/issue36707
+    if py_major <= 3 and py_minor < 8:
+        platform += "m"
     if (py_major, py_minor) == (2, 7):
         platform += "u"
     pex_request = TwoStepPexFromTargetsRequest(


Test patch: