Code change:
On 2019-10-24 we had a CI outage due to the ca-certificates msys2 package being broken. The hack implemented (:heart:) is to download an older, known good package beforehand and install it after msys2 updated its packages. Once is fixed we should revert the hack, but I think at this point it's worth investing in vendoring msys2 as a whole.
msys2 is using and other tools from archlinux distro. Arch Linux provides per-day official repositories snapshots. User can "pin" to a specific date. I think it is possible to reuse that mechanism to vendoring msys2 as a whole, by providing a mirror of a specific date's version. See here for more descriptions .
that would be nice but msys doesn't remove old packages from their repo so the mirror would be enormous. Search for in , only those packages take dozens of GiB. Also extracting it from archive would make CI a bit faster because you won't be modifying thousands of small files over and over. On Linux that is not a problem but working through cygwin emulation a bit struggles on this part.
No update on this, planning to discuss what to vendor at the all hands.
At the last infra meeting we decided a good step is to move towards using the pre-installed msys2 packages. In theory this is as simple as removing the msys2 installation step and pointing things at the right location on the images in CI.

Related discussion:
text: source "$(cd "$(dirname "$0")" && pwd)/../shared.sh" if isWindows; then <del> # Pre-followed the api/v2 URL to the CDN since the API can be a bit flakey curl -sSL https://packages.chocolatey.org/msys2.20190524.0.0.20191030.nupkg >  msys2.nupkg curl -sSL https://packages.chocolatey.org/chocolatey-core.extension.1.3.5.1.nupkg >  chocolatey-core.extension.nupkg choco install -s . msys2  --params="/InstallDir:$(ciCheckoutPath)/msys2 /NoPath" -y --no-progress rm msys2.nupkg chocolatey-core.extension.nupkg mkdir -p "$(ciCheckoutPath)/msys2/home/${USERNAME}" ciCommandAddPath "$(ciCheckoutPath)/msys2/usr/bin" </del> <ins> msys2Path="c:/msys64" mkdir -p "${msys2Path}/home/${USERNAME}" ciCommandAddPath "${msys2Path}/usr/bin" </ins> echo "switching shell to use our own bash" <del> ciCommandSetEnv CI_OVERRIDE_SHELL "$(ciCheckoutPath)/msys2/usr/bin/bash.exe" </del> <ins> ciCommandSetEnv CI_OVERRIDE_SHELL "${msys2Path}/usr/bin/bash.exe" # Detect the native Python version installed on the agent. On GitHub # Actions, the C:hostedtoolcachewindowsPython directory contains a # subdirectory for each installed Python version. # # The -V flag of the sort command sorts the input by version number. native_python_version="$(ls /c/hostedtoolcache/windows/Python | sort -Vr | head -n 1)" # Make sure we use the native python interpreter instead of some msys equivalent # one way or another. The msys interpreters seem to have weird path conversions # baked in which break LLVM's build system one way or another, so let's use the # native version which keeps everything as native as possible. python_home="/c/hostedtoolcache/windows/Python/${native_python_version}/x64" cp "${python_home}/python.exe" "${python_home}/python3.exe" ciCommandAddPath "C:hostedtoolcachewindowsPython${native_python_version}x64" ciCommandAddPath "C:hostedtoolcachewindowsPython${native_python_version}x64Scripts" </ins> fi