Repository: pypa/cibuildwheel

Issue:
Python 3.12?
### Description

Now that CPython 3.12.0b1 is out, I would like to produce wheels for it.  I know the cibuildwheel docs say to wait for rc1, but the advice from Brett Cannon and Thomas Wouters is that it is fine to ship wheels now as long as I produce new wheels in the unlikely event that the ABI changes.

I [tried specifying 3.12, but it failed](https://github.com/nedbat/coveragepy/actions/runs/5073139235/jobs/9111774802#step:6:72)
```
cibuildwheel: No build identifiers selected: BuildSelector(build_config='cp312-*', skip_config='', requires_python=<SpecifierSet('>=3.7')>, prerelease_pythons=True)
```

### Build log

https://github.com/nedbat/coveragepy/actions/runs/5073139235/jobs/9111774802#step:6:72

### CI config

https://github.com/nedbat/coveragepy/blob/nedbat/312-kits/.github/workflows/kit.yml
[Bot] Update dependencies
Update the versions of our dependencies.

PR generated by "Update dependencies" [workflow](https://github.com/pypa/cibuildwheel/actions/runs/5082190344).


Discussion:
(Pasted from the pypa discord, where I was replying to something related, but the contents roughly is of interest here.) Also I could have sworn that I responded that we have to wait till https://github.com/pypa/manylinux/pull/1483, which got merged yesterday. I forget the build frequency, but it's either weekly or daily.

----

manylinux has [roughly the same warning](https://github.com/pypa/manylinux/issues/1484). One key difference is a wheel built with an ABI difference will likely segfault, compared to a Python wheel that will produce a readable error. And the first 3.x binary is very important for NumPy, which will forever use that wheel to build all 3.12 wheels in the future (okay, patch releases could possibly be moved to, but never a minor version. And even going up a patch version has to be extensively tested in `oldest-supported-numpy`. So NumPy is likely the last possible (though I'm with some of the NumPy team this week, so I could ask to see what they think).

I think we need a guarantee that binary compatibility will be preserved before we can recommend building and uploading support for Python-beta wheels. Or a tag to indicate that wheels were made with a beta (<4) Python. If a project tests cross-beta release compatibility (that is, downloads the wheels again and tests them against the new betas) and immediately deletes them if there's an issue, it's probably fine. But then if you are doing that you probably know what you are doing and don't need advice.

It also takes some time for projects to test, so for even the projects that want to upload beta wheels, I'd probably at the minimum wait till beta 2. I think there's a ton of movement right before beta 1, so I'd not really consider relying on beta 1's stability to be safe.
Maybe I don't understand some of the factors here (likely!)...  I'm not counting on the ABI being stable. I'm gambling that it is likely to be stable, and accepting that if it changes, I will need to make new 3.12 wheels.  This is the advice from the Steering Council.

I don't understand why you are mentioning Numpy in your answer. If they have unusual requirements, can't they can decide their own path?

I get it that building wheels on beta releases can involve complications that wheels on stable releases don't have.  I'm asking for the ability to make those wheels and navigate those complications myself.
The short answer is that it'll be available under the CIBW_PRERELEASE_PYTHONS flag, probably within the next few days. 
Apologies, that made less sense out of context than I expected it to. The discussion was about the steering council recommendation clashing with cibuildwheel and manylinux's warning, which state pretty much the opposite.

> I'm gambling that it is likely to be stable, and accepting that if it changes, I will need to make new 3.12 wheels.

You can't upload a new Python 3.12 wheel, you only get to upload it once per release. You can delete the broken wheel and (ideally) make a new release with the updated 3.12 wheel, but many projects don't make releases based on fixing 3.12 wheels. If you know what to do, it's fine to do it, I was just defending the cibuildwheel/manylinux stance of making this available but with a warning and a custom flag, and this issue was a handy place to dump the text in a (more) public place. It absolutely will be available soon, and you absolutely can upload if you know what you are doing.
Just checked, the latest manylinux has `Python 3.12.0b1`!
Great, thanks for clearing that up.
> Just checked, the latest manylinux has `Python 3.12.0b1`!

I guess we need a new release of cibuildwheel for it to work?
> You can't upload a new Python 3.12 wheel, you only get to upload it once per release.

It's possible to upload a new wheel for a release via build numbers.

For example, we did it not once but twice in https://github.com/python-pillow/Pillow/issues/6750

https://pypi.org/project/Pillow/9.4.0/#files has:

* `Pillow-9.4.0-cp311-cp311-macosx_10_10_x86_64.whl`
* `Pillow-9.4.0-1-cp311-cp311-macosx_10_10_x86_64.whl`
* `Pillow-9.4.0-2-cp311-cp311-macosx_10_10_x86_64.whl`
That allows you to only upload one new wheel rather than the full set, right? I think I assumed it was the full set. That would help a lot (as long as you tracked failures and actually rebuilt wheels if there was one - and `wheel tags` would help).

I would like to know how common ABI breakages during betas are.
> That allows you to only upload one new wheel rather than the full set, right?

Yes, you can do it for as many or as few as you need. In this Pillow case, we specifically re-uploaded the x86-64 macOS wheels, as only those were broken.


Resolving patch:
diff --git a/cibuildwheel/util.py b/cibuildwheel/util.py
--- a/cibuildwheel/util.py
+++ b/cibuildwheel/util.py
@@ -251,7 +251,7 @@ class BuildSelector:
     requires_python: SpecifierSet | None = None
 
     # a pattern that skips prerelease versions, when include_prereleases is False.
-    PRERELEASE_SKIP: ClassVar[str] = ""
+    PRERELEASE_SKIP: ClassVar[str] = "cp312-*"
     prerelease_pythons: bool = False
 
     def __call__(self, build_id: str) -> bool:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -9,7 +9,7 @@
 
 nox.options.sessions = ["lint", "pylint", "check_manifest", "tests"]
 
-PYTHON_ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
+PYTHON_ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"]
 
 DIR = Path(__file__).parent.resolve()
 


Test patch:
diff --git a/test/test_abi_variants.py b/test/test_abi_variants.py
--- a/test/test_abi_variants.py
+++ b/test/test_abi_variants.py
@@ -47,7 +47,7 @@ def test_abi3(tmp_path):
     expected_wheels = [
         w.replace("cp38-cp38", "cp38-abi3")
         for w in utils.expected_wheels("spam", "0.1.0")
-        if "-pp" not in w and "-cp39" not in w and "-cp310" not in w and "-cp311" not in w
+        if "-pp" not in w and "-cp39" not in w and "-cp31" not in w
     ]
     assert set(actual_wheels) == set(expected_wheels)
 
diff --git a/test/test_dependency_versions.py b/test/test_dependency_versions.py
--- a/test/test_dependency_versions.py
+++ b/test/test_dependency_versions.py
@@ -120,10 +120,10 @@ def test_dependency_constraints_file(tmp_path, build_frontend_env):
     project_with_expected_version_checks.generate(project_dir)
 
     tool_versions = {
-        "pip": "20.0.2",
-        "setuptools": "53.0.0",
-        "wheel": "0.36.2",
-        "virtualenv": "20.11.2",
+        "pip": "23.1.2",
+        "setuptools": "67.7.2",
+        "wheel": "0.38.3",
+        "virtualenv": "20.23.0",
     }
 
     constraints_file = tmp_path / "constraints file.txt"
@@ -155,11 +155,12 @@ def test_dependency_constraints_file(tmp_path, build_frontend_env):
         add_env={
             "CIBW_ENVIRONMENT": cibw_environment_option,
             "CIBW_DEPENDENCY_VERSIONS": str(constraints_file),
+            "CIBW_SKIP": "cp36-*",
             **build_frontend_env,
         },
     )
 
     # also check that we got the right wheels
-    expected_wheels = utils.expected_wheels("spam", "0.1.0")
+    expected_wheels = [w for w in utils.expected_wheels("spam", "0.1.0") if "-cp36" not in w]
 
     assert set(actual_wheels) == set(expected_wheels)
diff --git a/test/test_manylinuxXXXX_only.py b/test/test_manylinuxXXXX_only.py
--- a/test/test_manylinuxXXXX_only.py
+++ b/test/test_manylinuxXXXX_only.py
@@ -84,8 +84,11 @@ def test(manylinux_image, tmp_path):
         # We don't have a manylinux1 image for PyPy & CPython 3.10 and above
         add_env["CIBW_SKIP"] = "pp* cp31*"
     if manylinux_image in {"manylinux2010"}:
-        # We don't have a manylinux2010 image for PyPy 3.9, CPython 3.11
-        add_env["CIBW_SKIP"] = "pp39* cp311*"
+        # We don't have a manylinux2010 image for PyPy 3.9, CPython 3.11+
+        add_env["CIBW_SKIP"] = "pp39* cp311* cp312*"
+    if manylinux_image in {"manylinux_2_24"}:
+        # We don't have a manylinux_2_24 image for CPython 3.12+
+        add_env["CIBW_SKIP"] = "cp312*"
     if manylinux_image == "manylinux_2_28" and platform.machine() == "x86_64":
         # We don't have a manylinux_2_28 image for i686
         add_env["CIBW_ARCHS"] = "x86_64"
@@ -109,7 +112,15 @@ def test(manylinux_image, tmp_path):
 
     if manylinux_image in {"manylinux2010"}:
         # remove PyPy 3.9 & CPython 3.11
-        expected_wheels = [w for w in expected_wheels if "-pp39" not in w and "-cp311" not in w]
+        expected_wheels = [
+            w
+            for w in expected_wheels
+            if "-pp39" not in w and "-cp311" not in w and "-cp312" not in w
+        ]
+
+    if manylinux_image in {"manylinux_2_24"}:
+        # remove CPython 3.11 and above
+        expected_wheels = [w for w in expected_wheels if "-cp312" not in w]
 
     if manylinux_image == "manylinux_2_28" and platform.machine() == "x86_64":
         # We don't have a manylinux_2_28 image for i686
diff --git a/test/test_pep518.py b/test/test_pep518.py
--- a/test/test_pep518.py
+++ b/test/test_pep518.py
@@ -9,7 +9,7 @@
         """
         # Will fail if PEP 518 does work
         import requests
-        assert requests.__version__ == "2.23.0", "Requests found but wrong version ({0})".format(requests.__version__)
+        assert requests.__version__ == "2.27.0", "Requests found but wrong version ({0})".format(requests.__version__)
 
         # Just making sure environment is still set
         import os
@@ -27,7 +27,7 @@
     "setuptools >= 42",
     "setuptools_scm[toml]>=4.1.2",
     "wheel",
-    "requests==2.23.0"
+    "requests==2.27.0"
 ]
 
 build-backend = "setuptools.build_meta"
diff --git a/test/utils.py b/test/utils.py
--- a/test/utils.py
+++ b/test/utils.py
@@ -170,6 +170,7 @@ def expected_wheels(
             "cp39-cp39",
             "cp310-cp310",
             "cp311-cp311",
+            "cp312-cp312",
         ]
 
         if machine_arch in ["x86_64", "AMD64", "x86", "aarch64"]:
@@ -182,6 +183,7 @@ def expected_wheels(
                 "cp39-cp39",
                 "cp310-cp310",
                 "cp311-cp311",
+                "cp312-cp312",
                 "pp38-pypy38_pp73",
                 "pp39-pypy39_pp73",
             ]
diff --git a/unit_test/build_selector_test.py b/unit_test/build_selector_test.py
--- a/unit_test/build_selector_test.py
+++ b/unit_test/build_selector_test.py
@@ -12,6 +12,7 @@ def test_build():
     assert build_selector("cp37-manylinux_x86_64")
     assert build_selector("cp310-manylinux_x86_64")
     assert build_selector("cp311-manylinux_x86_64")
+    assert not build_selector("cp312-manylinux_x86_64")
     assert build_selector("pp36-manylinux_x86_64")
     assert build_selector("pp37-manylinux_x86_64")
     assert build_selector("cp36-manylinux_i686")
@@ -31,6 +32,7 @@ def test_build():
     assert build_selector("cp37-win_amd64")
     assert build_selector("cp310-win_amd64")
     assert build_selector("cp311-win_amd64")
+    assert not build_selector("cp312-win_amd64")
     assert not build_selector("pp36-win_amd64")
     assert not build_selector("pp37-win_amd64")
 
@@ -43,9 +45,9 @@ def test_build_filter_pre():
     )
 
     assert build_selector("cp37-manylinux_x86_64")
-    assert build_selector("cp311-manylinux_x86_64")
+    assert build_selector("cp312-manylinux_x86_64")
     assert build_selector("cp37-win_amd64")
-    assert build_selector("cp311-win_amd64")
+    assert build_selector("cp312-win_amd64")
 
 
 def test_skip():