Repository: pyodide/pyodide-build
Pull request #167: Test against a minimum version of Pyodide, and document how to update it
Author: login: agriyakhetarpal
Created: 2025-04-01T20:11:36Z | Merged: 2025-04-02T19:57:26Z

## Description

Closes #149. This PR adds two new jobs in the CI matrix for recipe and src tests, respectively, that run against a minimum supported version of Pyodide that we specify in [the `pyodide-cross-build-environments.json` file](https://github.com/pyodide/pyodide/blob/74bd69b5afa00074580f16a72eae3d7ce5a0817a/pyodide-cross-build-environments.json). These run only on Linux for now. The result is that haven't broken compatibility with Pyodide 0.26 xbuildenvs so far.

Brief instructions have also been added to address breaking changes upstream in the Pyodide repository, coupled with general release instructions.

Linked issue #149: Test against a minimum version of Pyodide
We currently have a minimum supported Pyodide version, but we don't test against it. This means that we aren't aware when to bump that value up.

I think we should introduce a new CI job (Linux only and maybe running with a `[min-pyodide]` commit marker) which should test this, and update the cross-build environment metadata file if we find that we have broken compatibility with that version.
Linked issue comments: user: login: ryanking13
created_at: 2025-03-28T08:34:18Z
body: Sounds good to me. Maybe we can update the integration test to test with the min-pyodide version and the tot-pyodide (unless it introduces too many jobs).
author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-01T19:48:02Z
body: Looking into this more, it indeed makes sense to not run the unit tests but rather the integration tests only. As we already have a commit marker for running them, I'll skip the extra commit marker for these additional integration tests in this case. There's no way to configure the default xbuildenv version for the unit tests until #158 is merged, anyway.
author_association: MEMBER

Diff:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 00b786ea..5b183d6a 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -13,7 +13,7 @@ concurrency:
 env:
   FORCE_COLOR: 3
   # Increase this value to reset cache if emscripten_version has not changed
-  EMSDK_CACHE_FOLDER: 'emsdk-cache'
+  EMSDK_CACHE_FOLDER: "emsdk-cache"
   EMSDK_CACHE_NUMBER: 0
 
 jobs:
@@ -90,19 +90,30 @@ jobs:
           fi
 
   integration-test:
+    name: integration-test (${{ matrix.task.name }}, ${{ matrix.task.installer }}, ${{ matrix.pyodide-version }}-pyodide, ${{ matrix.os }})
     runs-on: ${{ matrix.os }}
     needs: [check-integration-test-trigger]
     strategy:
       fail-fast: false
       matrix:
         task: [
-          {name: test-recipe, installer: pip},
-          {name: test-src, installer: pip},
-          {name: test-recipe, installer: uv},
-          {name: test-src, installer: uv},
-          {name: test-integration-marker, installer: pip},  # installer doesn't matter
-        ]
+            { name: test-recipe, installer: pip },
+            { name: test-src, installer: pip },
+            { name: test-recipe, installer: uv },
+            { name: test-src, installer: uv },
+            { name: test-integration-marker, installer: pip }, # installer doesn't matter
+          ]
         os: [ubuntu-latest, macos-latest]
+        pyodide-version: [stable]
+        # Run Pyodide minimum version testing only for the pip installer and on Linux
+        include:
+          - task: { name: test-recipe, installer: pip }
+            os: ubuntu-latest
+            pyodide-version: minimum
+          - task: { name: test-src, installer: pip }
+            os: ubuntu-latest
+            pyodide-version: minimum
+
     if: needs.check-integration-test-trigger.outputs.run-integration-test
     steps:
       - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
@@ -127,7 +138,13 @@ jobs:
 
       - name: Install xbuildenv
         run: |
-          pyodide xbuildenv install
+          if [[ "${{ matrix.pyodide-version }}" == "minimum" ]]; then
+            MIN_VERSION=$(pyodide xbuildenv search --json | jq -r '.environments | sort_by(.version) | .[0].version')
+            echo "Installing Pyodide xbuildenv version: $MIN_VERSION"
+            pyodide xbuildenv install $MIN_VERSION
+          else
+            pyodide xbuildenv install
+          fi
           echo EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version) >> $GITHUB_ENV
 
       - name: Cache emsdk
@@ -150,7 +167,7 @@ jobs:
         if: matrix.task.name == 'test-integration-marker'
         run: pytest --junitxml=test-results/junit.xml --cov=pyodide-build pyodide_build -m "integration"
 
-      - name: Run the recipe integration tests (${{ matrix.task }})
+      - name: Run the recipe integration tests (${{ matrix.task.name }})
         if: matrix.task.name != 'test-integration-marker'
         env:
           PYODIDE_JOBS: ${{ steps.get-cores.outputs.CORES }}
diff --git a/pyodide_build/recipe/builder.py b/pyodide_build/recipe/builder.py
index 506b74dd..c2798988 100755
--- a/pyodide_build/recipe/builder.py
+++ b/pyodide_build/recipe/builder.py
@@ -596,7 +596,14 @@ def _package_wheel(
             )
             if self.build_metadata.cross_build_env:
                 subprocess.run(
-                    ["pip", "install", "-t", str(host_site_packages), f"{name}=={ver}"],
+                    [
+                        "pip",
+                        "install",
+                        "--upgrade",
+                        "-t",
+                        str(host_site_packages),
+                        f"{name}=={ver}",
+                    ],
                     check=True,
                 )
 
diff --git a/release.md b/release.md
new file mode 100644
index 00000000..847e8c6b
--- /dev/null
+++ b/release.md
@@ -0,0 +1,26 @@
+# `pyodide-build` release workflow
+
+This document provides maintenance and release instructions for `pyodide-build`, adapted
+from the [Maintainer information page from the Pyodide project](https://pyodide.org/en/stable/development/maintainers.html).
+
+> [!NOTE]
+> `pyodide-build` is used as a git submodule by the Pyodide repository. If a change is only required for in-tree builds,
+it is sufficient to update the git submodule. A release is only necessary if the changes are required by out-of-tree builds.
+>
+> See the [Updating `pyodide-build`](https://pyodide.org/en/stable/development/maintainers.html#updating-pyodide-build) section for more information.
+
+# Release instructions
+
+- Decide on a new version number: we follow the [SemVer](https://semver.org/) versioning scheme; which means that we create major/minor versions for feature releases and micro/patch versions for bug fixes. The package uses [Semantic Versioning](https://semver.org/) for versioning. The version number is defined in the `pyproject.toml` file in the root of the repository.
+
+- If there is a compelling reason to discuss or plan a new release before creating it (what is to be included, timeline, planned scope, etc.), open a new "Release planning" tracking issue with the information and any other relevant resources. Optionally, a new milestone for the release can also be created. This is not strictly necessary, but it can help to keep track of the issues and PRs that are planned to be included.
+
+- Create a tag with the version number (e.g., `v0.X.Y`) while on the `main` branch, and create and publish a GitHub release with the tag and attach a link to the CHANGELOG. Other release notes can be generated with the "Generate release notes" button in the GitHub UI, which will add a list of all PRs that were merged with the tag. The release notes can be reviewed and edited as necessary at the time of creating the release or edited after it is created. The release notes from the previous releases can be used for reference and inspiration.
+
+- If the release is introducing breaking changes are introduced that will not be compatible with cross-build environments for older versions of Pyodide:
+    - the release notes should be updated to include a "Breaking changes" section, if not already present through the CHANGELOG.
+    - we also test against a minimum version of Pyodide xbuildenvs in the integration tests in `main.yml`. If a new minimum version
+    is required, then:
+        - update the value of the `MIN_COMPATIBLE_PYODIDE_BUILD_VERSION` variable in the [`tools/update_cross_build_releases.py`](https://github.com/pyodide/pyodide/blob/74bd69b5afa00074580f16a72eae3d7ce5a0817a/tools/update_cross_build_releases.py#L23-L25) file, and
+        - update the `maximum_pyodide_build_version` variable accordingly in the [`pyodide-cross-build-environments.json`](https://github.com/pyodide/pyodide/blob/bc82c83c27527a8e384474df880be6a290fcfbb1/pyodide-cross-build-environments.json) file,
+    so that the cross-build environments metadata for newer versions of Pyodide will contain the correct minimum and maximum versions of `pyodide-build` they can be used with.


Reviews:
user: login: hoodmane
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T12:45:42Z
user: login: hoodmane
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T12:49:35Z
user: login: hoodmane
body: I'm okay to merge as-is and we can discuss any changes to `release.md` in a followup.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-02T12:51:18Z
user: login: ryanking13
body: Looks good to me except for minor comments. Thanks for working on this!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-02T13:49:16Z
user: login: ryanking13
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T14:01:19Z
user: login: hoodmane
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T15:21:47Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T18:45:53Z

Review comments:
diff_hunk: @@ -0,0 +1,27 @@
+# pyodide-build release workflow
+
+This document provides maintenance and release instructions for `pyodide-build` project, adapted
+from the [Maintainer information page from the Pyodide project](https://pyodide.org/en/stable/development/maintainers.html)
+
+> [!NOTE]
+> `pyodide-build` is also used as a submodule in-tree within the Pyodide repository, and it is usually acceptable to
+update the submodule to point to a newer commit in most cases and a release isn't always required unless it's needed
+to address bugs or add features for out-of-tree builds.
path: release.md
user: login: hoodmane
body: ```suggestion
> `pyodide-build` is used as a git submodule by the Pyodide repository. If a change is only required for in-tree builds,
it is sufficient to update the git submodule. A release is only necessary if the changes are required by out-of-tree builds. 
```
created_at: 2025-04-02T12:45:41Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,27 @@
+# pyodide-build release workflow
+
+This document provides maintenance and release instructions for `pyodide-build` project, adapted
+from the [Maintainer information page from the Pyodide project](https://pyodide.org/en/stable/development/maintainers.html)
+
+> [!NOTE]
+> `pyodide-build` is also used as a submodule in-tree within the Pyodide repository, and it is usually acceptable to
+update the submodule to point to a newer commit in most cases and a release isn't always required unless it's needed
+to address bugs or add features for out-of-tree builds.
+>
+> See the [Updating `pyodide-build](https://pyodide.org/en/stable/development/maintainers.html#updating-pyodide-build) for more information.
+
+# Release instructions
+
+- First, create a "Release planning" tracking issue that can list information such as:
+    - the timeline of the release
+    - the scope of the release and the pull requests to be included, ideally by attaching a milestone to them or creating one through the GitHub Issues UI if it doesn't exist
+    - any other relevant resources
+
+- Once a consensus is reached on the release plan, create a new milestone in the GitHub Issues UI. This milestone should have the same name as the release version (e.g., `v0.23.0`) and should be marked as "open". The milestone should also be assigned to the release planning issue.
path: release.md
user: login: hoodmane
body: I'd expect this amount of process will be unnecessary. Ideally if any single maintainer feels like making a release for any reason they should be able to just do it, unless there is some compelling reason to have more process. Releases are small and they should be easy to make as long as we have faith in our CI pipelines. If something is missing, we can just make a followup release.

I think the only thing that we might need to discuss is whether a release should be major or minor. I think it would be good to have a policy here on how we decide that.
created_at: 2025-04-02T12:49:35Z
author_association: MEMBER
diff_hunk: @@ -127,7 +138,13 @@ jobs:
 
       - name: Install xbuildenv
         run: |
-          pyodide xbuildenv install
+          if [[ "${{ matrix.pyodide-version }}" == "minimum" ]]; then
+            MIN_VERSION=$(pyodide xbuildenv search --json | jq -r '.environments | sort_by(.version) | .[0].version')
path: .github/workflows/main.yml
user: login: ryanking13
body: Nice. Thanks again for making `--json` option by the way.
created_at: 2025-04-02T13:39:24Z
author_association: MEMBER
diff_hunk: @@ -90,19 +90,30 @@ jobs:
           fi
 
   integration-test:
+    name: integration-test (${{ matrix.task.name }}, ${{ matrix.task.installer }}, ${{ matrix.pyodide-version }}-pyodide, ${{ matrix.os }})
     runs-on: ${{ matrix.os }}
     needs: [check-integration-test-trigger]
     strategy:
       fail-fast: false
       matrix:
         task: [
-          {name: test-recipe, installer: pip},
-          {name: test-src, installer: pip},
-          {name: test-recipe, installer: uv},
-          {name: test-src, installer: uv},
-          {name: test-integration-marker, installer: pip},  # installer doesn't matter
-        ]
+            { name: test-recipe, installer: pip },
+            { name: test-src, installer: pip },
+            { name: test-recipe, installer: uv },
+            { name: test-src, installer: uv },
+            { name: test-integration-marker, installer: pip }, # installer doesn't matter
+          ]
         os: [ubuntu-latest, macos-latest]
+        pyodide-version: [stable]
path: .github/workflows/main.yml
user: login: ryanking13
body: (no action required for this PR)

For a reproducible test, maybe we should specify the pyodide version tested in the integration test somewhere, instead of letting pyodide-build grab the last compatible version automatically 
created_at: 2025-04-02T13:40:57Z
author_association: MEMBER
diff_hunk: @@ -596,7 +596,14 @@ def _package_wheel(
             )
             if self.build_metadata.cross_build_env:
                 subprocess.run(
-                    ["pip", "install", "-t", str(host_site_packages), f"{name}=={ver}"],
+                    [
+                        "pip",
+                        "install",
+                        "--upgrade",
path: pyodide_build/recipe/builder.py
user: login: ryanking13
body: Just curious why we need an `--upgrade` here. I think passing the version explicitly would reinstall the package anyway if there is different version installed.
created_at: 2025-04-02T13:43:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,27 @@
+# pyodide-build release workflow
+
+This document provides maintenance and release instructions for `pyodide-build` project, adapted
+from the [Maintainer information page from the Pyodide project](https://pyodide.org/en/stable/development/maintainers.html)
+
+> [!NOTE]
+> `pyodide-build` is also used as a submodule in-tree within the Pyodide repository, and it is usually acceptable to
+update the submodule to point to a newer commit in most cases and a release isn't always required unless it's needed
+to address bugs or add features for out-of-tree builds.
+>
+> See the [Updating `pyodide-build](https://pyodide.org/en/stable/development/maintainers.html#updating-pyodide-build) for more information.
+
+# Release instructions
+
+- First, create a "Release planning" tracking issue that can list information such as:
+    - the timeline of the release
+    - the scope of the release and the pull requests to be included, ideally by attaching a milestone to them or creating one through the GitHub Issues UI if it doesn't exist
+    - any other relevant resources
+
+- Once a consensus is reached on the release plan, create a new milestone in the GitHub Issues UI. This milestone should have the same name as the release version (e.g., `v0.23.0`) and should be marked as "open". The milestone should also be assigned to the release planning issue.
path: release.md
user: login: ryanking13
body: +1.

If there isn't a breaking change, I think a maintainer can make a release when necessary. That's the main reason why we made a separate pyodide-build repository.

I think we can make an issue when we have a reason to block the release until some fixes are merged.
created_at: 2025-04-02T13:47:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,27 @@
+# `pyodide-build` release workflow
+
+This document provides maintenance and release instructions for `pyodide-build`, adapted
+from the [Maintainer information page from the Pyodide project](https://pyodide.org/en/stable/development/maintainers.html).
+
+> [!NOTE]
+> `pyodide-build` is also used as a submodule in-tree within the Pyodide repository, and it is usually acceptable to
+update the submodule to point to a newer commit in most cases and a release isn't always required unless it's needed
+to address bugs or add features for out-of-tree builds.
+>
+> See the [Updating `pyodide-build`](https://pyodide.org/en/stable/development/maintainers.html#updating-pyodide-build) section for more information.
+
+# Release instructions
+
+- First, create a "Release planning" tracking issue that can list information such as:
+    - the timeline of the release
+    - the scope of the release and the pull requests to be included, ideally by attaching a milestone to them or creating one through the GitHub Issues UI if it doesn't exist
+    - any other relevant resources
+
+- Once a consensus is reached on the release plan, create a new milestone in the GitHub Issues UI. This milestone should have the same name as the release version (e.g., `v0.23.0`) and should be marked as "open". The milestone should also be assigned to the release planning issue.
+
+- Create a tag with the version number (e.g., `v0.X.Y`) while on the `main` branch, and create and publish a GitHub release with the tag and attach a link to the CHANGELOG. Other release notes can be generated with the "Generate release notes" button, which will add a list of all PRs that were merged with the tag. The release notes can be reviewed and edited as necessary at the time of creating the release or edited after it is created.
+
+- If backward-incompatible changes are introduced:
+    - the release notes should be updated to include a "Breaking changes" section, if not already present through the CHANGELOG.
+    - we also test against a minimum version of Pyodide xbuildenvs in the integration tests in `main.yml`. If a new minimum version
+    is required, then update it in the [`tools/update_cross_build_releases.py`](https://github.com/pyodide/pyodide/blob/74bd69b5afa00074580f16a72eae3d7ce5a0817a/tools/update_cross_build_releases.py#L23-L25) file so that cross-build environments metadata for newer versions of Pyodide will contain the correct minimum and maximum versions of `pyodide-build` they can be used with.
path: release.md
user: login: ryanking13
body: 👍  Thanks for writing this. To be more precise, if we make a breaking change in pyodide-build that is not compatible to the old Pyodide versions, one should

1. update the `MIN_COMPATIBLE_PYODIDE_BUILD_VERSION ` in the update_cross_build_releases.py file.
2. update the `maximum_pyodide_build_version` in the `pyodide-cross-build-environments.json` for all the old pyodide versions that are not compatible with the latest pyodide-build release.
created_at: 2025-04-02T14:01:19Z
author_association: MEMBER
diff_hunk: @@ -596,7 +596,14 @@ def _package_wheel(
             )
             if self.build_metadata.cross_build_env:
                 subprocess.run(
-                    ["pip", "install", "-t", str(host_site_packages), f"{name}=={ver}"],
+                    [
+                        "pip",
+                        "install",
+                        "--upgrade",
path: pyodide_build/recipe/builder.py
user: login: hoodmane
body: One would think. I think this is a pip bug with the `-t` option, it actually adds the `numpy-2.0.2-dist-info` directory but leaves the `numpy-1.3.2-dist-info` directory too and doesn't update the package.
created_at: 2025-04-02T15:21:47Z
author_association: MEMBER
diff_hunk: @@ -596,7 +596,14 @@ def _package_wheel(
             )
             if self.build_metadata.cross_build_env:
                 subprocess.run(
-                    ["pip", "install", "-t", str(host_site_packages), f"{name}=={ver}"],
+                    [
+                        "pip",
+                        "install",
+                        "--upgrade",
path: pyodide_build/recipe/builder.py
user: login: agriyakhetarpal
body: I think it is the intended behaviour. Here is what the `-t` option from `pip install --help` says:

```
Install packages into <dir>. By default this will not replace existing files/folders in
<dir>. Use --upgrade to replace existing packages in <dir> with new versions.
```
created_at: 2025-04-02T18:45:53Z
author_association: MEMBER

PR comments:
user: login: agriyakhetarpal
created_at: 2025-04-02T00:26:21Z
body: > [!TIP]
> The issue described here has been fixed with e3b7e9a90f0ab1801550f5f2447b8ffd723e5f0f; however, suggestions/feedback on the rest of the thread are welcome.

We're apparently no longer compatible with Pyodide 0.26 – at least for recipe builds. Here are my thoughts:

- The workflow run in https://github.com/pyodide/pyodide-build/actions/runs/14206005642/job/39803823812?pr=167 was successful in building NumPy for WASM with Pyodide 0.26.0's xbuildenv, but it then downloads a native NumPy wheel as a part of the post-build step for some unexplained reason and tries to operate on it (instead of modifying the NumPy WASM wheel it built), and it then subsequently fails in the process. The rest of the recipe builds are passing as they don't have a post-build step.
- I'm able to reproduce this failure with NumPy locally, both on `main` and with the v0.29.3 tag. However, I'm unable to reproduce it with other packages. I tried a post-build script with a recipe for SymPy from its sdist (as it is pure Python), and then with `zfpy` (as it's a compiled package depending on a static library), and both passed without issues.
- The error persists even if I remove the post-build step from NumPy – it's unable to find the NumPy headers, as the folder containing them was renamed.
- This suggests that the issue is:
	- specific to NumPy as it's included in the cross-build environments, and
	- not specific to post-build steps for NumPy, as removing the `rm -rf numpy/distutils/__pycache__` step still makes the build fail
	- specific to the NumPy and the Pyodide version, as Pyodide 0.27 and therefore NumPy 2.0 changed where the headers are located in the package's structure

I have not checked whether this issue will exist for any other packages wherein we set `cross-build-env: true` and provide `cross-build-files:` for.

Then, I tried `pyodide-build` v0.29.3 as a user for an out-of-tree build for NumPy, where it passes (on Linux; there's an unexpected and unrelated error around `libm` on macOS).

@hoodmane and @ryanking13, am I right in understanding that:
- the xbuildenv was mainly introduced and is more or less intended for out-of-tree builds for end users, rather than for recipe builds for Pyodide developers/contributors (which are highly tailored for the Pyodide repository at the moment),
- as long as the out-of-tree builds are working, we don't need to bump the minimum supported Pyodide version in the cross-build environments metadata file just yet?
- as a result, we should test only the src builds with a minimum Pyodide version and not the recipe builds?

Another point to note in this PR is that we are testing the tip-of-tree version of `pyodide-build` with the CI mechanism I configured here, and not a tagged version of `pyodide-build`. So, this means that once we encounter a problem we can't work around in terms of providing backward compatibility when working on a PR, we have to bump the version in the cross-build environments metadata file in the main Pyodide repo.

A question I have about this is: does the "latest" word in line 23 in https://github.com/pyodide/pyodide/blob/74bd69b5afa00074580f16a72eae3d7ce5a0817a/tools/update_cross_build_releases.py#L23-L25 mean we can update the minimum value as soon as we merge a PR here where we break backward compatibility, or we need to do so when we create a `pyodide-build` release that includes such a PR? My reason for asking is that we could think this doesn't matter as Pyodide and pyodide-build are being released separately anyway, but there is always a chance we miss out on updating that file for whatever reason. So, perhaps it would make sense if I were to add a small "Release process for maintainers" document via this PR that writes some basic steps to go through when creating a `pyodide-build` release?

author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-02T00:27:15Z
body: Marking it as ready for review for now despite the failing build, I'll edit the PR description based on what comes out of our discussion here.
author_association: MEMBER
user: login: hoodmane
created_at: 2025-04-02T08:41:58Z
body: > it then downloads a native NumPy wheel as a part of the post-build step for some unexplained reason and tries to operate on it (instead of modifying the NumPy WASM wheel it built)

It needs to produce an xbuild wheel. Downstream packages that use numpy at build time need a manylinux numpy whose libs and headers are the wasm libs and headers.

I think we should fix this. I think the problem is related to the fact that numpy 2.0 moved the locations of these files?
author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-02T08:57:00Z
body: > > it then downloads a native NumPy wheel as a part of the post-build step for some unexplained reason and tries to operate on it (instead of modifying the NumPy WASM wheel it built)
> 
> It needs to produce an xbuild wheel. Downstream packages that use numpy at build time need a manylinux numpy whose libs and headers are the wasm libs and headers.

Ah, I see. I thought that downloading a manylinux/native wheel was a weird quirk!

> I think we should fix this. I think the problem is related to the fact that numpy 2.0 moved the locations of these files?

Yes
author_association: MEMBER
user: login: hoodmane
created_at: 2025-04-02T10:44:18Z
body: I think I fixed it.
author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-02T11:54:30Z
body: Thanks a lot, @hoodmane!
author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-02T11:56:45Z
body: I added some brief release instructions in a10701f94d3f596fd222fcb2f9d18868f3992374. If they look good to you, then we should be good to go and merge this.
author_association: MEMBER
user: login: hoodmane
created_at: 2025-04-02T12:50:17Z
body: Thanks for working on this @agriyakhetarpal! This is very helpful.
author_association: MEMBER
user: login: ryanking13
created_at: 2025-04-02T13:57:09Z
body: > the xbuildenv was mainly introduced and is more or less intended for out-of-tree builds for end users, rather than for recipe builds for Pyodide developers/contributors (which are highly tailored for the Pyodide repository at the moment).
> as long as the out-of-tree builds are working, we don't need to bump the minimum supported Pyodide version in the cross-build environments metadata file just yet?
> as a result, we should test only the src builds with a minimum Pyodide version and not the recipe builds?

I think recipe builds are also something we'll continue to use, so if there's a breaking change affecting many packages, I think it's a good to call it an “incompatible version”.

However, I also think it's a bit harsh to call something incompatible just because it breaks one or two specific packages. If the problem with numpy is a problem with all packages that use numpy as a build dependency, then we can call it incompatible, but if it's just numpy itself, then I think it's fine.

> So, this means that once we encounter a problem we can't work around in terms of providing backward compatibility when working on a PR, we have to bump the version in the cross-build environments metadata file in the main Pyodide repo.

I don't quite get what this means. If we break something in pyodide-build, we should fix in in tip-of-tree pyodide-build anyway isn't it?


author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-02T19:13:01Z
body: > I'd expect this amount of process will be unnecessary. Ideally if any single maintainer feels like making a release for any reason they should be able to just do it, unless there is some compelling reason to have more process. Releases are small and they should be easy to make as long as we have faith in our CI pipelines. If something is missing, we can just make a followup release.
> 
> I think the only thing that we might need to discuss is whether a release should be major or minor. I think it would be good to have a policy here on how we decide that.


> +1.
> 
> If there isn't a breaking change, I think a maintainer can make a release when necessary. That's the main reason why we made a separate pyodide-build repository.
> 
> I think we can make an issue when we have a reason to block the release until some fixes are merged.

Thanks for your comments here. I agree with both of these points; honestly, I intended to document a formal process here, but I wasn't expecting that either of us would follow it 😂 We don't want to be bureaucratic in our release procedure, so I'll tone it down. Having the provision to test things out in the main Pyodide repo is also useful, and we have all made changes recently which are then tested before merging, and are caught in CI or in reviews.

The only valid reason for having a slightly stricter release process, IMO, is that it is easy for us to push broken to PyPI and break downstream packages that are using non-pinned versions of `pyodide-build`. The integration tests largely help with this, but gaps in our testing will always exist, as is the case with all software.

As for a policy on how releases, i.e., version bumps are decided: we are following SemVer already, so a feature will land as a new minor release, and bug fixes will land in micro releases.
author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-02T19:22:03Z
body: > > the xbuildenv was mainly introduced and is more or less intended for out-of-tree builds for end users, rather than for recipe builds for Pyodide developers/contributors (which are highly tailored for the Pyodide repository at the moment).
> > as long as the out-of-tree builds are working, we don't need to bump the minimum supported Pyodide version in the cross-build environments metadata file just yet?
> > as a result, we should test only the src builds with a minimum Pyodide version and not the recipe builds?
> 
> I think recipe builds are also something we'll continue to use, so if there's a breaking change affecting many packages, I think it's a good to call it an “incompatible version”.
> 
> However, I also think it's a bit harsh to call something incompatible just because it breaks one or two specific packages. If the problem with numpy is a problem with all packages that use numpy as a build dependency, then we can call it incompatible, but if it's just numpy itself, then I think it's fine.

Thanks for the explanation, and yes, that is a fair point! One failure here feels a bit more prominent and important to fix, as it's a failure in one out of seven packages being tested – but it doesn't feel the same way in the Pyodide repo as it would be, say, one failure out of ~240 packages. I'm used to the CI being flaky there, but not here. I'll keep the minimum Pyodide version tests for recipe and src builds.

> > So, this means that once we encounter a problem we can't work around in terms of providing backward compatibility when working on a PR, we have to bump the version in the cross-build environments metadata file in the main Pyodide repo.
> 
> I don't quite get what this means. If we break something in pyodide-build, we should fix in in tip-of-tree pyodide-build anyway isn't it?

Ah, I'm sorry if I might have been too terse here! What I was confused about was the timing of when we need to update the metadata file in the Pyodide repository: either at the time when we introduce a backward-incompatible feature in `pyodide-build`, or when we release that feature in a new `pyodide-build` version. You answered this in https://github.com/pyodide/pyodide-build/pull/167#discussion_r2024896771. Either way, if there is a feature that we think will be incompatible with older Pyodide versions, we can always test it by temporarily updating the submodule.


author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-02T19:56:50Z
body: Addressed all review comments except the one on making the test more reproducible by hardcoding what the minimum supported version is – I'll open a new issue about it. Thanks for the reviews!
author_association: MEMBER
user: login: hoodmane
created_at: 2025-04-02T21:08:06Z
body: Thanks again @agriyakhetarpal !
author_association: MEMBER