Repository: pypa/cibuildwheel
Pull request #2414: Debug Cirrus CI testing
Author: login: joerick
Labels: name: CI: PyPy
description: Run the integration test suite with PyPy included
name: CI: GraalPy
description: Run the integration test suite with GraalPy included
Created: 2025-05-25T17:23:01Z | Merged: 2025-05-29T14:52:46Z

Fix the build on `main`. 

## Disables iOS testing on Cirrus.

Cirrus has a problem running the simulator, see a recent build on main, for example:

https://cirrus-ci.com/task/6394806932340736?logs=run_cibuildwheel_tests#L1832

I've removed the support tick from cirrus in the readme too, until/unless we can figure this out. 

cc @freakboy3742

## Fixes a bug with the GraalPy build

Sometimes vsdevcmd can print an error to console while it works. This messed up the environment JSON blob, so it's saved to a file instead.



Diff:
diff --git a/README.md b/README.md
index 8d260da70..ee342659e 100644
--- a/README.md
+++ b/README.md
@@ -63,7 +63,7 @@ Usage
 | Travis CI       | ✅    |       | ✅      | ✅        |           |             |     |
 | CircleCI        | ✅    | ✅    |         | ✅        | ✅        |             | ✅³  |
 | Gitlab CI       | ✅    | ✅    | ✅      | ✅¹       | ✅        |             | ✅³  |
-| Cirrus CI       | ✅    | ✅    | ✅      | ✅        | ✅        |             | ✅³  |
+| Cirrus CI       | ✅    | ✅    | ✅      | ✅        | ✅        |             |      |
 
 <sup>¹ [Requires emulation](https://cibuildwheel.pypa.io/en/stable/faq/#emulation), distributed separately. Other services may also support Linux ARM through emulation or third-party build hosts, but these are not tested in our CI.</sup><br>
 <sup>² [Uses cross-compilation](https://cibuildwheel.pypa.io/en/stable/faq/#windows-arm64). It is not possible to test `arm64` on this CI platform.</sup><br>
diff --git a/cibuildwheel/platforms/windows.py b/cibuildwheel/platforms/windows.py
index 17375fad4..819914a00 100644
--- a/cibuildwheel/platforms/windows.py
+++ b/cibuildwheel/platforms/windows.py
@@ -336,37 +336,36 @@ def setup_python(
         # variables. Adapted from
         # https://github.com/microsoft/vswhere/wiki/Start-Developer-Command-Prompt
         # Remove when https://github.com/oracle/graalpython/issues/492 is fixed.
-        vcpath = subprocess.check_output(
-            [
-                Path(os.environ["PROGRAMFILES(X86)"])
-                / "Microsoft Visual Studio"
-                / "Installer"
-                / "vswhere.exe",
-                "-products",
-                "*",
-                "-latest",
-                "-property",
-                "installationPath",
-            ],
-            text=True,
+        vcpath = call(
+            Path(os.environ["PROGRAMFILES(X86)"])
+            / "Microsoft Visual Studio"
+            / "Installer"
+            / "vswhere.exe",
+            "-products",
+            "*",
+            "-latest",
+            "-property",
+            "installationPath",
+            capture_stdout=True,
         ).strip()
         log.notice(f"Discovering Visual Studio for GraalPy at {vcpath}")
-        vcvars = subprocess.check_output(
-            [
-                f"{vcpath}\\Common7\\Tools\\vsdevcmd.bat",
-                "-no_logo",
-                "-arch=amd64",
-                "-host_arch=amd64",
-                "&&",
-                "python",
-                "-c",
-                "import os, json, sys; json.dump(dict(os.environ), sys.stdout);",
-            ],
-            shell=True,
-            text=True,
+        vcvars_file = tmp / "vcvars.json"
+        call(
+            f"{vcpath}\\Common7\\Tools\\vsdevcmd.bat",
+            "-no_logo",
+            "-arch=amd64",
+            "-host_arch=amd64",
+            "&&",
+            "python",
+            "-c",
+            # this command needs to be one line for Windows reasons
+            "import sys, json, pathlib, os; pathlib.Path(sys.argv[1]).write_text(json.dumps(dict(os.environ)))",
+            vcvars_file,
             env=env,
         )
-        env.update(json.loads(vcvars))
+        with open(vcvars_file, encoding="utf-8") as f:
+            vcvars = json.load(f)
+        env.update(vcvars)
 
     return base_python, env
 
diff --git a/test/test_ios.py b/test/test_ios.py
index 9ee588b26..3d1674585 100644
--- a/test/test_ios.py
+++ b/test/test_ios.py
@@ -8,6 +8,8 @@
 
 import pytest
 
+from cibuildwheel.ci import CIProvider, detect_ci_provider
+
 from . import test_projects, utils
 
 basic_project_files = {
@@ -23,6 +25,19 @@ def test_platform(self):
 }
 
 
+def skip_if_ios_testing_not_supported() -> None:
+    """Skip the test if iOS testing is not supported on this machine."""
+    if utils.get_platform() != "macos":
+        pytest.skip("this test can only run on macOS")
+    if utils.get_xcode_version() < (13, 0):
+        pytest.skip("this test only works with Xcode 13.0 or greater")
+    if detect_ci_provider() == CIProvider.cirrus_ci:
+        pytest.skip(
+            "iOS testing not currently supported on Cirrus CI due to a failure "
+            "to start the simulator."
+        )
+
+
 # iOS tests shouldn't be run in parallel, because they're dependent on calling
 # Xcode, and starting a simulator. These are both multi-threaded operations, and
 # it's easy to overload the CI machine if there are multiple test processes
@@ -39,10 +54,7 @@ def test_platform(self):
     ],
 )
 def test_ios_platforms(tmp_path, build_config, monkeypatch, capfd):
-    if utils.get_platform() != "macos":
-        pytest.skip("this test can only run on macOS")
-    if utils.get_xcode_version() < (13, 0):
-        pytest.skip("this test only works with Xcode 13.0 or greater")
+    skip_if_ios_testing_not_supported()
 
     # Create a temporary "bin" directory, symlink a tool that we know eixsts
     # (/usr/bin/true) into that location under a name that should be unique,
@@ -94,10 +106,7 @@ def test_ios_platforms(tmp_path, build_config, monkeypatch, capfd):
 @pytest.mark.serial
 def test_no_test_sources(tmp_path, capfd):
     """Build will provide a helpful error if pytest is run and test-sources is not defined."""
-    if utils.get_platform() != "macos":
-        pytest.skip("this test can only run on macOS")
-    if utils.get_xcode_version() < (13, 0):
-        pytest.skip("this test only works with Xcode 13.0 or greater")
+    skip_if_ios_testing_not_supported()
 
     project_dir = tmp_path / "project"
     basic_project = test_projects.new_c_project()
@@ -126,10 +135,7 @@ def test_no_test_sources(tmp_path, capfd):
 
 def test_ios_testing_with_placeholder(tmp_path, capfd):
     """Build will run tests with the {project} placeholder."""
-    if utils.get_platform() != "macos":
-        pytest.skip("this test can only run on macOS")
-    if utils.get_xcode_version() < (13, 0):
-        pytest.skip("this test only works with Xcode 13.0 or greater")
+    skip_if_ios_testing_not_supported()
 
     project_dir = tmp_path / "project"
     basic_project = test_projects.new_c_project()
@@ -155,10 +161,7 @@ def test_ios_testing_with_placeholder(tmp_path, capfd):
 
 def test_missing_xbuild_tool(tmp_path, capfd):
     """Build will fail if xbuild-tools references a non-existent tool."""
-    if utils.get_platform() != "macos":
-        pytest.skip("this test can only run on macOS")
-    if utils.get_xcode_version() < (13, 0):
-        pytest.skip("this test only works with Xcode 13.0 or greater")
+    skip_if_ios_testing_not_supported()
 
     project_dir = tmp_path / "project"
     basic_project = test_projects.new_c_project()
@@ -183,10 +186,7 @@ def test_missing_xbuild_tool(tmp_path, capfd):
 
 def test_no_xbuild_tool_definition(tmp_path, capfd):
     """Build will succeed with a warning if there is no xbuild-tools definition."""
-    if utils.get_platform() != "macos":
-        pytest.skip("this test can only run on macOS")
-    if utils.get_xcode_version() < (13, 0):
-        pytest.skip("this test only works with Xcode 13.0 or greater")
+    skip_if_ios_testing_not_supported()
 
     project_dir = tmp_path / "project"
     basic_project = test_projects.new_c_project()
@@ -220,10 +220,7 @@ def test_no_xbuild_tool_definition(tmp_path, capfd):
 
 def test_empty_xbuild_tool_definition(tmp_path, capfd):
     """Build will succeed with no warning if there is an empty xbuild-tools definition."""
-    if utils.get_platform() != "macos":
-        pytest.skip("this test can only run on macOS")
-    if utils.get_xcode_version() < (13, 0):
-        pytest.skip("this test only works with Xcode 13.0 or greater")
+    skip_if_ios_testing_not_supported()
 
     project_dir = tmp_path / "project"
     basic_project = test_projects.new_c_project()
@@ -255,10 +252,7 @@ def test_empty_xbuild_tool_definition(tmp_path, capfd):
 @pytest.mark.serial
 def test_ios_test_command_without_python_dash_m(tmp_path, capfd):
     """pytest should be able to run without python -m, but it should warn."""
-    if utils.get_platform() != "macos":
-        pytest.skip("this test can only run on macOS")
-    if utils.get_xcode_version() < (13, 0):
-        pytest.skip("this test only works with Xcode 13.0 or greater")
+    skip_if_ios_testing_not_supported()
 
     project_dir = tmp_path / "project"
 
@@ -296,10 +290,7 @@ def test_spam():
 
 def test_ios_test_command_invalid(tmp_path, capfd):
     """Test command should raise an error if it's clearly invalid."""
-    if utils.get_platform() != "macos":
-        pytest.skip("this test can only run on macOS")
-    if utils.get_xcode_version() < (13, 0):
-        pytest.skip("this test only works with Xcode 13.0 or greater")
+    skip_if_ios_testing_not_supported()
 
     project_dir = tmp_path / "project"
     basic_project = test_projects.new_c_project()


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR fixes CI test failures by disabling iOS testing on Cirrus CI, streamlines skip logic in the iOS tests, and improves the GraalPy environment capture process on Windows. It also updates the README to reflect Cirrus CI's unsupported iOS testing status.

- Centralize and apply `skip_if_ios_testing_not_supported` in `test/test_ios.py`
- Change GraalPy setup to write environment JSON to a file in `cibuildwheel/platforms/windows.py`
- Remove Cirrus CI iOS support indicator in `README.md`

### Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

| File                                    | Description                                                      |
|-----------------------------------------|------------------------------------------------------------------|
| test/test_ios.py                        | Added `skip_if_ios_testing_not_supported` helper and replaced inline skips, disabling iOS tests on Cirrus CI |
| cibuildwheel/platforms/windows.py       | Capture GraalPy environment via a temporary JSON file instead of `check_output` |
| README.md                               | Cleared the Cirrus CI support tick to indicate unsupported iOS testing |




state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T11:30:13Z
user: login: henryiii
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T14:51:58Z

Review comments:
diff_hunk: @@ -63,7 +63,7 @@ Usage
 | Travis CI       | ✅    |       | ✅      | ✅        |           |             |     |
 | CircleCI        | ✅    | ✅    |         | ✅        | ✅        |             | ✅³  |
 | Gitlab CI       | ✅    | ✅    | ✅      | ✅¹       | ✅        |             | ✅³  |
-| Cirrus CI       | ✅    | ✅    | ✅      | ✅        | ✅        |             | ✅³  |
+| Cirrus CI       | ✅    | ✅    | ✅      | ✅        | ✅        |             |      |
path: README.md
user: login: Copilot
body: [nitpick] The empty cell for Cirrus CI iOS testing in the README table may be ambiguous; consider using an explicit '❌' or adding a footnote to clarify that iOS testing is not supported on Cirrus CI.
created_at: 2025-05-29T11:30:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,37 +336,36 @@ def setup_python(
         # variables. Adapted from
         # https://github.com/microsoft/vswhere/wiki/Start-Developer-Command-Prompt
         # Remove when https://github.com/oracle/graalpython/issues/492 is fixed.
-        vcpath = subprocess.check_output(
-            [
-                Path(os.environ["PROGRAMFILES(X86)"])
-                / "Microsoft Visual Studio"
-                / "Installer"
-                / "vswhere.exe",
-                "-products",
-                "*",
-                "-latest",
-                "-property",
-                "installationPath",
-            ],
-            text=True,
+        vcpath = call(
+            Path(os.environ["PROGRAMFILES(X86)"])
+            / "Microsoft Visual Studio"
+            / "Installer"
+            / "vswhere.exe",
+            "-products",
+            "*",
+            "-latest",
+            "-property",
+            "installationPath",
+            capture_stdout=True,
         ).strip()
         log.notice(f"Discovering Visual Studio for GraalPy at {vcpath}")
-        vcvars = subprocess.check_output(
-            [
-                f"{vcpath}\\Common7\\Tools\\vsdevcmd.bat",
-                "-no_logo",
-                "-arch=amd64",
-                "-host_arch=amd64",
-                "&&",
-                "python",
-                "-c",
-                "import os, json, sys; json.dump(dict(os.environ), sys.stdout);",
-            ],
-            shell=True,
-            text=True,
+        vcvars_file = tmp / "vcvars.json"
+        call(
+            f"{vcpath}\\Common7\\Tools\\vsdevcmd.bat",
+            "-no_logo",
+            "-arch=amd64",
+            "-host_arch=amd64",
+            "&&",
+            "python",
+            "-c",
+            # this command needs to be one line for Windows reasons
+            "import sys, json, pathlib, os; pathlib.Path(sys.argv[1]).write_text(json.dumps(dict(os.environ)))",
path: cibuildwheel/platforms/windows.py
user: login: Copilot
body: [nitpick] The long inline Python one-liner is hard to read and maintain; consider extracting this snippet into a named constant or separate script for clarity.
```suggestion
            f"python -c \"{_generate_vcvars_command()}\"",
```
created_at: 2025-05-29T11:30:13Z
author_association: CONTRIBUTOR

PR comments:
user: login: freakboy3742
created_at: 2025-05-25T22:50:18Z
body: AFAICT, this is a manifestation of python/cpython#133131 - Xcode removed the iPhone SE simulator from the default set when they released Xcode 16.3, in favor of the newly released "iPhone 16e". As a result, the default simulator picked by the iOS testbed runner no longer exists. 

This has been fixed upstream, and the patch will be rolled into the next release of the iOS binary support package. I was planning to cut an update once 3.14 beta 2 is released (which is due in a day or two).
author_association: CONTRIBUTOR
user: login: henryiii
created_at: 2025-05-27T21:55:29Z
body: Beta 2 is out now! Also waiting on @mayut to trigger manylinux.
author_association: CONTRIBUTOR
user: login: freakboy3742
created_at: 2025-05-28T05:06:26Z
body: > Beta 2 is out now! Also waiting on @mayut to trigger manylinux.

ACK - still trying to dig my way out of my Pycon US backlog. Will cut a new release as soon as I surface :-)
author_association: CONTRIBUTOR
user: login: joerick
created_at: 2025-05-28T09:51:26Z
body: I think you meant @mayeut above!
author_association: CONTRIBUTOR
user: login: joerick
created_at: 2025-05-28T16:18:18Z
body: Whomp whomp, ran out of cirrus credits. I'm pretty sure i figured out what was wrong, vsdevcmd was outputting an error message that was corrupting the JSON stream.  I'll tidy this up.
author_association: CONTRIBUTOR
user: login: joerick
created_at: 2025-05-29T11:28:51Z
body: This is in fact, ready, the Cirrus CI failures are due to lack of credits.
author_association: CONTRIBUTOR