Repository: pypa/cibuildwheel

Issue:
3.11 builds failing with v2.8.1 on manylinux2010 images
### Description

With:
```yaml
        cibw_python: [ "cp37-*", "cp38-*", "cp39-*", "cp310-*", "cp311-*"]
```
and
```yaml
        env:
          CIBW_PRERELEASE_PYTHONS: True
```

I'm seeing this error:
```
Setting up build environment...
  
      + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
      + which python
  cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.
  Error: Process completed with exit code 1.
```

This didn't occur with 2.8.0 - I suspect it's somehow related to this changelog entry:

> 🛠 The GitHub Action will ensure a compatible version of Python is installed on the runner (#1114)

I'm also seeing
```
Run actions/setup-python@v4
  with:
    python-version: 3.7 - 3.10
    update-environment: false
    token: ***
  env:
    CIBW_BUILD: cp311-*
    CIBW_ARCHS_LINUX: x86_64
    CIBW_MANYLINUX_X86_64_IMAGE: manylinux[20](https://github.com/bloomberg/memray/runs/7420878603?check_suite_focus=true#step:3:21)10
    CIBW_MANYLINUX_I686_IMAGE: manylinux2010
    CIBW_PRERELEASE_PYTHONS: true
    CIBW_TEST_EXTRAS: test
    CIBW_TEST_COMMAND: pytest {package}/tests
```
in the build log - perhaps that "python-version: 3.7 - 3.10" is the problem?

### Build log

https://github.com/bloomberg/memray/runs/7421778954?check_suite_focus=true

### CI config

https://github.com/bloomberg/memray/blob/f9bfaacea95c9c565189da2eeb121a3303c11c71/.github/workflows/build_wheels.yml


Discussion:
That’s not what that range means. It uses one version from that, and the outer host setting shouldn’t matter. I wish we printed the details of the mismatch. We are using 3.11b4 now. Don’t see how that would change this either, though. Am traveling and can’t check quite yet.
Why is it using `/opt/python/cp38-cp38/bin/python`? Host is 3.10 and target is 3.11. Don’t get where 3.8 is sneaking in.
I'm past my depth in debugging this, but if there's anything I can run to help you make sense of it, I'd be happy to. I've added a `type -a python` to `before-all` and it prints out:
```
  python is /opt/python/cp38-cp38/bin/python
  python is /usr/bin/python
```
but I'm not sure where the `/opt/python/cp38-cp38/bin/python` is coming from...

Likewise, printing `$PATH` is showing me
```
  PATH=/opt/python/cp38-cp38/bin:/opt/rh/devtoolset-8/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
```

This could be something I've got misconfigured, but v2.8.1 is failing in a way that v2.8.0 didn't...
manylinux2010 does not have CPython 3.11
support for this image will be dropped before ABI stability of CPython 3.11 thus the beta has been dropped on this image to prevent building 3.11 with a Python that will never be ABI stable on that image.
You'll have to use manylinux2014 image to build CPython 3.11
- https://github.com/pypa/manylinux/issues/1281
- https://github.com/pypa/manylinux/commit/97c4ab0bc6f0d9a44f60e7938b025c69788a89b7
Oof. OK... Well, that's not good news for me, since I do still need RHEL 6 compatibility, but at least that answers what changed.

Thanks for helping me look into this.
Thanks for the debugging. We could use a better error message for this then. @mayeut I'm guessing manylinux1 also won't get Python 3.11?
We need to detect this then, rather than grabbing a random Python. :)

manylinux1 is “dead”, it’s just still being built while CI doesn’t break, so it’s not getting anything new. Not sure you can build 3.11 on CentOS 5.

You can use an override to build 3.11 with manylinux2014. In fact, IIRC, NumPy switched to 2014 for 3.10, since anyone on 3.10 should be on a 2014 compatible platform. Doesn’t help to have higher compatibility than your dependencies.
> I'm guessing manylinux1 also won't get Python 3.11?

Correct. The manylinux1 image does not even have Python 3.10

> We need to detect this then, rather than grabbing a random Python. :)

It's correctly detected. The error message "cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it." relates to that (could be clearer by checking the path added to PATH exists in the first place).

The `/opt/python/cp38-cp38/bin/python` python mentioned is the one used as a build driver inside the container (counterpart to the host python running cibuildwheel).


Yep, it is detected, it's just the error message is confusing because that message is assuming that the user has fiddled with PATH to break it, not that it never existed in the image.

We can probably add a different message, like `if not python_bin.exists(): raise Exception(f'Python {python_configuration.version} doesn't exist in image {image}')`. And even perhaps a fail-fast before the builds start.
> The `/opt/python/cp38-cp38/bin/python` python mentioned is the one used as a build driver inside the container (counterpart to the host python running cibuildwheel).

I don't recall adding this to PATH in cibuildwheel. Perhaps it's added in manylinux?
> Perhaps it's added in manylinux?

The only `python` in the `PATH` in manylinux images is the system one (when it exists, on some images the system python is `python3`)

I probably wasn't clear enough about `/opt/python/cp38-cp38/bin/python`.
> The `/opt/python/cp38-cp38/bin/python` python mentioned is the one used as a build driver inside the container (counterpart to the host python running cibuildwheel).

It's almost always used by cibuildwheel with the full file path: https://github.com/pypa/cibuildwheel/blob/7c7dda72084bd9203b4fffd3af856560f8e48c64/cibuildwheel/oci_container.py#L46
except for before_all step:
https://github.com/pypa/cibuildwheel/blob/7c7dda72084bd9203b4fffd3af856560f8e48c64/cibuildwheel/linux.py#L128-L137

ahh.. gotcha. So the cp38 in PATH thing really is a red herring. It's only a result of trying to investigate this using before-all.

Resolving patch:
diff --git a/cibuildwheel/linux.py b/cibuildwheel/linux.py
--- a/cibuildwheel/linux.py
+++ b/cibuildwheel/linux.py
@@ -110,6 +110,29 @@ def get_build_steps(
     yield from steps.values()
 
 
+def check_all_python_exist(
+    *, platform_configs: list[PythonConfiguration], container: OCIContainer
+) -> None:
+    exist = True
+    messages = []
+    for config in platform_configs:
+        python_path = config.path / "bin" / "python"
+        try:
+            container.call(["test", "-x", python_path])
+        except subprocess.CalledProcessError:
+            messages.append(
+                f"  '{python_path}' executable doesn't exist in image '{container.image}' to build '{config.identifier}'."
+            )
+            exist = False
+    if not exist:
+        message = "\n".join(messages)
+        print(
+            f"cibuildwheel:\n{message}",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+
 def build_in_container(
     *,
     options: Options,
@@ -120,6 +143,8 @@ def build_in_container(
 ) -> None:
     container_output_dir = PurePosixPath("/output")
 
+    check_all_python_exist(platform_configs=platform_configs, container=container)
+
     log.step("Copying project into container...")
     container.copy_into(Path.cwd(), container_project_path)
 


Test patch:
diff --git a/test/test_linux_python.py b/test/test_linux_python.py
new file mode 100644
--- /dev/null
+++ b/test/test_linux_python.py
@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+import platform
+import subprocess
+
+import pytest
+
+from . import test_projects, utils
+
+
+def test_python_exist(tmp_path, capfd):
+    if utils.platform != "linux":
+        pytest.skip("the test is only relevant to the linux build")
+    machine = platform.machine()
+    if machine not in ["x86_64", "i686"]:
+        pytest.skip(
+            "this test is currently only possible on x86_64/i686 due to availability of alternative images"
+        )
+
+    project_dir = tmp_path / "project"
+    basic_project = test_projects.new_c_project()
+    basic_project.generate(project_dir)
+
+    with pytest.raises(subprocess.CalledProcessError):
+        utils.cibuildwheel_run(
+            project_dir,
+            add_env={
+                "CIBW_MANYLINUX_X86_64_IMAGE": "manylinux2010",
+                "CIBW_MANYLINUX_I686_IMAGE": "manylinux2010",
+                "CIBW_BUILD": "cp3{10,11}-manylinux*",
+            },
+        )
+
+    captured = capfd.readouterr()
+    print("out", captured.out)
+    print("err", captured.err)
+    assert f" to build 'cp310-manylinux_{machine}'." not in captured.err
+    message = (
+        "'/opt/python/cp311-cp311/bin/python' executable doesn't exist"
+        f" in image 'quay.io/pypa/manylinux2010_{machine}:2022-08-05-4535177'"
+        f" to build 'cp311-manylinux_{machine}'."
+    )
+    assert message in captured.err