Repository: tahoe-lafs/zfec (language: python)

Issue:
Update GA workflow actions
Splitting #75 into three separate issues. This is part 1:

- Many workflow actions are considered outdated because of https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.
- Since v1. 9.0, cibuildwheel itself has a GitHub Action (https://cibuildwheel.readthedocs.io/en/stable/changelog/#v190), using which we can reduce some boilerplate.
- OS versions used in test and build workflows have diverged.
- Relatedly, maybe we should not be using GitHub's interpretation of `-latest` versions of OSes (which can change over time, although with enough warning), and should be more explicit about which version of an OS we're using: `ubuntu-22.04` instead of `ubuntu-latest` etc.  Since zfec development moves at a slower rate than GitHub's development, we will not get a surprise breakage when occasionally revisiting zfec.
Test and build with newer Pythons
See #74.  These are the changes:

- Add Python 3.11 and PyPy 3.9 to both test and build matrices.
- Update cibuildwheel version (because of the above), and use cibuildwheel action rather than the Python package (because it avoids some of the boilerplate).
- Use the same OSes in test and build steps, and be more explicit about them (not just `-latest`).
- Build macOS arm64 and universal2 wheels as well (because why not), but without testing them, because cibuildwheels docs [says](https://cibuildwheel.readthedocs.io/en/stable/options/) that they can't be tested on Intel. ([Results](https://test.pypi.org/project/zfec/1.5.7.2.post0.dev21/#files) are on TestPyPI.)
- Update the soon-to-be deprecated actions.
- Cache Python packages when possible.
add Python 3.11 support
It works fine for me with 3.10.6, but with python 3.11 it doesn't install. Edit: see exarkuns comment. I did change the Issue name.
Build macOS arm64 or/and universal2 wheels
Splitting off #75 : 

Now that non-Intel Apple hardware has been a thing for a while, and it would be useful to build arm64 and universal2 (or maybe both?) wheels. Seems that cibuildwheel can build all of them using GitHub Workflows.

On a new MacBook, `pip install -i https://test.pypi.org/simple/ zfec==1.5.7.2.post0.dev21` installed the arm64 binary; on an older MacBook it installed the x86_64 binary. Not sure if we'd need universal2 binary.


Discussion / hints:

> See #74. These are the changes:
> 
>     * Add Python 3.11 and PyPy 3.9 to both test and build matrices.
> 
>     * Update cibuildwheel version (because of the above), and use cibuildwheel action rather than the Python package (because it avoids some of the boilerplate).
> 
>     * Use the same OSes in test and build steps, and be more explicit about them (not just `-latest`).
> 
>     * Build macOS arm64 and universal2 wheels as well (because why not), but without testing them, because cibuildwheels docs [says](https://cibuildwheel.readthedocs.io/en/stable/options/) that they can't be tested on Intel. ([Results](https://test.pypi.org/project/zfec/1.5.7.2.post0.dev21/#files) are on TestPyPI.)
> 
>     * Update the soon-to-be deprecated actions.
> 
>     * Cache Python packages when possible.

Thanks.  Do you mind splitting this into a few different PRs?  At a glance, it seems like three makes sense:

1. CI housekeeping - action updates & caching improvements
2. Build universal wheels
3. Add more Python versions to the build/test matrices

How does that sound?
> Thanks. Do you mind splitting this into a few different PRs? At a glance, it seems like three makes sense:
> 
>     1. CI housekeeping - action updates & caching improvements
> 
>     2. Build universal wheels
> 
>     3. Add more Python versions to the build/test matrices
> 
> 
> How does that sound?

That sounds reasonable but I will have to watch the CI spin across three separate PRs again and that is not... fun. >_<

Will get on that when I have some time.
This is expected.  zfec is not yet tested against Python 3.11.  This ticket could be retitled to "add Python 3.11 support" and someone could do the necessary CI setup, fix any issues, and make a release happen.

@MironBodewes It may interest you that `zfex`, a fork of `zfec` which I maintain, builds and installs with cpython 3.11.
Did a quick test on macOS (arm64) and Fedora 37 (x86_64). With Python 3.11, `pip install zfec` does install from the sdist package when a C compiler is available, so the issue must be that there are no wheel packages for Python 3.11?

I suppose can can do some things:

- Add Python 3.10 and Python 3.11 to the "test" workflow.
- Update cibuildwheel in the "build" workflow (they're now at 2.x releases, and we're pinned at 1.6.4) and until it magically makes py310 and py311 wheels.  There's also a `pypa/cibuildwheel@version` action now, which simplifies things.

We could also look into abi3 wheels again (#49), now that cibuildwheel has support for it.


Fix patch:
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index bdd5febc..01c5292c 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -23,21 +23,21 @@ on:
   release:
     types: [published, created, edited]
 
-env:
-  # Just make sure that Python can use zfec package
-  CIBW_TEST_COMMAND: python -c "import zfec; print(zfec.__version__)"
 
 jobs:
   build_wheels:
     name: "${{ matrix.wheel-selector }} wheel on ${{ matrix.os }}"
     runs-on: ${{ matrix.os }}
+
     strategy:
       fail-fast: false
+      # NOTE: when updating the matrix below, be sure to keep OS and
+      # Python versions in sync with the ones in test.yml.
       matrix:
         os:
-          - "ubuntu-18.04"
-          - "windows-latest"
           - "macos-11"
+          - "ubuntu-22.04"
+          - "windows-2022"
         wheel-selector:
           - "cp37-*"
           - "cp38-*"
@@ -48,33 +48,25 @@ jobs:
 
     steps:
       - name: Check out zfec sources
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
         with:
           # Check out the full history, including tags.  This is necessary to
           # construct non-release version numbers.
           fetch-depth: 0
 
-      - uses: actions/setup-python@v2
-        name: Install Python
+      - name: Build wheels
+        uses: pypa/cibuildwheel@v2.3.1
         with:
-          python-version: '3.7'
-
-      - name: Install cibuildwheel
-        run: |
-          python3 -m pip install cibuildwheel==2.3.1
-          python -m cibuildwheel --print-build-identifiers
-
-      - name: "Run cibuildwheel"
+          output-dir: wheelhouse
         env:
           # Configure cibuildwheel to build just some of the total wheels we
           # could build for this architecture.  This yields better parallelism
           # on GitHub Actions execution.
           CIBW_BUILD: "${{ matrix.wheel-selector }}"
+          # Just make sure that Python can use zfec package.
+          CIBW_TEST_COMMAND: python -c "import zfec; print(zfec.__version__)"
 
-        run: |
-          python3 -m cibuildwheel --output-dir wheelhouse
-
-      - uses: actions/upload-artifact@v2
+      - uses: actions/upload-artifact@v3
         name: Upload artifacts
         with:
           path: ./wheelhouse/*.whl
@@ -84,23 +76,23 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         name: Check out zfec sources
         with:
           # Check out the full history, including tags.  This is necessary to
           # construct non-release version numbers.
           fetch-depth: 0
 
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v4
         name: Install Python
         with:
-          python-version: '3.7'
+          python-version: '3.11'
 
       - name: Build sdist
         run: |
           python3 setup.py sdist
 
-      - uses: actions/upload-artifact@v2
+      - uses: actions/upload-artifact@v3
         name: Upload artifacts
         with:
           path: dist/*.tar.gz
@@ -123,7 +115,7 @@ jobs:
       # Download all artifacts previously built by this workflow to the dist
       # directory where the publish step can find them.  These are the sdist
       # and all of the wheels build by the other jobs.
-      - uses: "actions/download-artifact@v2"
+      - uses: "actions/download-artifact@v3"
         with:
           name: "artifact"
           path: "dist"
@@ -138,7 +130,7 @@ jobs:
       #
       # The conditional in this step should cause it to run only for case (3).
       - name: "Publish to TEST PyPI"
-        uses: "pypa/gh-action-pypi-publish@master"
+        uses: "pypa/gh-action-pypi-publish@v1.6.4"
         if: >-
           github.event_name == 'pull_request'
 
@@ -160,7 +152,7 @@ jobs:
       # instance.  This time, we have a conditional that runs only for case
       # (2).
       - name: "Publish to LIVE PyPI"
-        uses: "pypa/gh-action-pypi-publish@master"
+        uses: "pypa/gh-action-pypi-publish@v1.6.4"
         if: >-
           github.event_name == 'push' &&
           startsWith(github.event.ref, 'refs/tags/zfec-')


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index eda6612d..0feafa0d 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -10,13 +10,16 @@ jobs:
 
   test:
     runs-on: ${{ matrix.os }}
+
     strategy:
       fail-fast: false
+      # NOTE: when updating the matrix below, be sure to keep OS and
+      # Python versions in sync with the ones in build.yml.
       matrix:
         os:
-          - macos-10.15
-          - ubuntu-latest
-          - windows-latest
+          - "macos-11"
+          - "ubuntu-22.04"
+          - "windows-2022"
         python-version:
           - "3.7"
           - "3.8"
@@ -28,12 +31,14 @@ jobs:
     steps:
 
       - name: Check out zfec sources
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
+          cache: 'pip' # caching pip dependencies
+          cache-dependency-path: 'setup.py'
 
       - name: "Run unit tests"
         run: |