Repository: bytecodealliance/wasmtime
Pull request #8612 (2024-05-14 01:40:54)

CI: shard testing crates across multiple jobs
<!--
Please make sure you include the following information:

- If this work has been discussed elsewhere, please include a link to that
  conversation. If it was discussed in an issue, just mention "issue #...".

- Explain why this change is needed. If the details are in an issue already,
  this can be brief.

Our development process is documented in the Wasmtime book:
https://docs.wasmtime.dev/contributing-development-process.html

Please ensure all communication follows the code of conduct:
https://github.com/bytecodealliance/wasmtime/blob/main/CODE_OF_CONDUCT.md
-->


fitzgen: Successful default PR run in ~8 minutes now: https://github.com/bytecodealliance/wasmtime/actions/runs/9073195225

Going to do a full run in a second.
fitzgen: Unfortunately, it looks like `cargo nextest`'s CI partitioning isn't exposed/supported by `cargo miri`.
fitzgen: Successful full test run in 14 minutes: https://github.com/bytecodealliance/wasmtime/actions/runs/9075621159?pr=8612
github-actions[bot]: #### Subscribe to Label Action

cc @cfallin, @fitzgen

<details>
This issue or pull request has been labeled: "cranelift", "cranelift:area:aarch64", "cranelift:area:machinst", "cranelift:area:x64", "isle", "wasmtime:api"

Thus the following users have been cc'd because of the following labels:

* cfallin: isle
* fitzgen: isle

To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.

[Learn more.](https://github.com/bytecodealliance/subscribe-to-label-action)
</details>
alexcrichton: Thanks for working on this! I'm personally a bit worried about the complexity creep here. Our CI is already pretty nontrivial and adding this into the mix I feel like takes it up to another level of complexity which makes it harder for future contributors/modifications. I can't argue with the results of course, but the merge queue currently reports that the average time-to-merge is 19 minutes so this whould shave 5 minutes off merge time. Given that we generally have an empty queue and we don't have many time-pressured PRs I'm not sure whether the 25% reduction in CI will be worth the complexity increase here.

I'll admit though I'm pretty biased as I'm still sort of used to the 3-hour-cycle-time from rust-lang/rust so my perception of what's appropriate for CI is likely pretty skewed. I'm curious how others feel as well as to the tradeoff of CI complexity here vs the time reduction.
fitzgen: Yeah I can definitely add more docs/comments if we want to continue down this route.

I think the default PR check time dropping from ~15 minutes to ~8 minutes is the real benefit here; more so than the 20 to 14 for full CI, at least IMO.
fitzgen: > I'll also point out that this is quite a lot of sharding, more than I've ever seen done before on github actions, so while I have no reason to think things will go wrong there's always the possibility we uncover something strange in the future. Most projects I don't think approach anywhere near this concurrency. The micro_checks job is also very heavily sharded where commands that might take ~5 seconds now take ~1 minute due to the overhead of a runner, downloading rustc, downloading deps, etc. That's not going to break the bank with our concurrency limit though so I don't have any reason to not do this.

Yeah I actually wanted to group that CI job into buckets of ~10 checks each or something but I couldn't figure out an easy way to do that ¯\\\_(ツ)\_/¯ 
fitzgen: Okay I think I figured out how to avoid having a billion `check` jobs
github-actions[bot]: #### Subscribe to Label Action

cc @fitzgen

<details>
This issue or pull request has been labeled: "fuzzing"

Thus the following users have been cc'd because of the following labels:

* fitzgen: fuzzing

To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.

[Learn more.](https://github.com/bytecodealliance/subscribe-to-label-action)
</details>

Patch:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index d5c1801a7..df6b167cd 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -299,12 +299,67 @@ jobs:
       env:
         GH_TOKEN: ${{ github.token }}
 
-  # Checks of various feature combinations and whether things
-  # compile. The goal here isn't to run tests, mostly just serve as a
-  # double-check that Rust code compiles and is likely to work everywhere else.
-  checks:
+  # Checks of various feature combinations and whether things compile. The goal
+  # here isn't to run tests, mostly just serve as a double-check that Rust code
+  # compiles and is likely to work everywhere else.
+  micro_checks:
     needs: determine
-    name: Check
+    name: Check ${{matrix.name}}
+    strategy:
+      fail-fast: true
+      matrix:
+        include:
+          - name: wasmtime
+            checks: |
+              -p wasmtime --no-default-features
+              -p wasmtime --no-default-features --features wat
+              -p wasmtime --no-default-features --features profiling
+              -p wasmtime --no-default-features --features cache
+              -p wasmtime --no-default-features --features async
+              -p wasmtime --no-default-features --features pooling-allocator
+              -p wasmtime --no-default-features --features cranelift
+              -p wasmtime --no-default-features --features component-model
+              -p wasmtime --no-default-features --features runtime,component-model
+              -p wasmtime --no-default-features --features cranelift,wat,async,cache
+              -p wasmtime --no-default-features --features winch
+              -p wasmtime --no-default-features --features wmemcheck
+              -p wasmtime --no-default-features --features wmemcheck,cranelift,runtime
+              -p wasmtime --no-default-features --features demangle
+              -p wasmtime --no-default-features --features addr2line
+              -p wasmtime --no-default-features --features gc
+              -p wasmtime --no-default-features --features runtime,gc
+              -p wasmtime --no-default-features --features cranelift,gc
+              -p wasmtime --no-default-features --features runtime
+              -p wasmtime --no-default-features --features threads
+              -p wasmtime --no-default-features --features runtime,threads
+              -p wasmtime --no-default-features --features cranelift,threads
+              -p wasmtime --features incremental-cache
+              -p wasmtime --all-features
+
+          - name: wasmtime-cli
+            checks: |
+              -p wasmtime-cli --no-default-features
+              -p wasmtime-cli --no-default-features --features run
+              -p wasmtime-cli --no-default-features --features run,component-model
+              -p wasmtime-cli --no-default-features --features compile
+              -p wasmtime-cli --no-default-features --features compile,cranelift
+              -p wasmtime-cli --no-default-features --features compile,cranelift,component-model
+              -p wasmtime-cli --all-features
+              -p wasmtime-cli --features component-model
+
+          - name: cranelift-codegen benches
+            checks: |
+              --benches -p cranelift-codegen
+
+          - name: wasmtime-bench-api
+            checks: |
+              -p wasmtime-bench-api
+
+          - name: wasmtime-c-api
+            checks: |
+              -p wasmtime-c-api --no-default-features
+              -p wasmtime-c-api --no-default-features --features wat
+              -p wasmtime-c-api --no-default-features --features wasi
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v4
@@ -312,52 +367,34 @@ jobs:
         submodules: true
     - uses: ./.github/actions/install-rust
 
-    # Check some feature combinations of the `wasmtime` crate
-    - run: cargo check -p wasmtime --no-default-features
-    - run: cargo check -p wasmtime --no-default-features --features wat
-    - run: cargo check -p wasmtime --no-default-features --features profiling
-    - run: cargo check -p wasmtime --no-default-features --features cache
-    - run: cargo check -p wasmtime --no-default-features --features async
-    - run: cargo check -p wasmtime --no-default-features --features pooling-allocator
-    - run: cargo check -p wasmtime --no-default-features --features cranelift
-    - run: cargo check -p wasmtime --no-default-features --features component-model
-    - run: cargo check -p wasmtime --no-default-features --features runtime,component-model
-    - run: cargo check -p wasmtime --no-default-features --features cranelift,wat,async,cache
-    - run: cargo check -p wasmtime --no-default-features --features winch
-    - run: cargo check -p wasmtime --no-default-features --features wmemcheck
-    - run: cargo check -p wasmtime --no-default-features --features wmemcheck,cranelift,runtime
-    - run: cargo check -p wasmtime --no-default-features --features demangle
-    - run: cargo check -p wasmtime --no-default-features --features addr2line
-    - run: cargo check -p wasmtime --no-default-features --features gc
-    - run: cargo check -p wasmtime --no-default-features --features runtime,gc
-    - run: cargo check -p wasmtime --no-default-features --features cranelift,gc
-    - run: cargo check -p wasmtime --no-default-features --features runtime
-    - run: cargo check -p wasmtime --no-default-features --features threads
-    - run: cargo check -p wasmtime --no-default-features --features runtime,threads
-    - run: cargo check -p wasmtime --no-default-features --features cranelift,threads
-    - run: cargo check --features component-model
-    - run: cargo check -p wasmtime --features incremental-cache
-    - run: cargo check -p wasmtime --all-features
-
-    # Feature combinations of the `wasmtime-cli`
-    - run: cargo check -p wasmtime-cli --no-default-features
-    - run: cargo check -p wasmtime-cli --no-default-features --features run
-    - run: cargo check -p wasmtime-cli --no-default-features --features run,component-model
-    - run: cargo check -p wasmtime-cli --no-default-features --features compile
-    - run: cargo check -p wasmtime-cli --no-default-features --features compile,cranelift
-    - run: cargo check -p wasmtime-cli --no-default-features --features compile,cranelift,component-model
-    - run: cargo check -p wasmtime-cli --all-features
-
-    # Check that benchmarks of the cranelift project build
-    - run: cargo check --benches -p cranelift-codegen
-
-    # Check that the bench-api compiles
-    - run: cargo check -p wasmtime-bench-api
-
-    # Check some feature combinations of the `wasmtime-c-api` crate
-    - run: cargo check -p wasmtime-c-api --no-default-features
-    - run: cargo check -p wasmtime-c-api --no-default-features --features wat
-    - run: cargo check -p wasmtime-c-api --no-default-features --features wasi
+    # Run the check.
+    - run: |
+        checks=$(cat <<END
+        ${{ matrix.checks }}
+        END
+        )
+        echo "$checks" | xargs -I CHECK sh -c 'echo "=== cargo check CHECK ==="; cargo check CHECK'
+
+    # Common logic to cancel the entire run if this job fails.
+    - run: gh run cancel ${{ github.run_id }}
+      if: failure() && github.event_name != 'pull_request'
+      env:
+        GH_TOKEN: ${{ github.token }}
+
+  # Similar to `micro_checks` but where we need to install some more state
+  # (e.g. Android NDK) and we haven't factored support for those things out into
+  # a parallel jobs yet.
+  monolith_checks:
+    needs: determine
+    name: Monolith Checks
+    runs-on: ubuntu-latest
+    env:
+      CARGO_NDK_VERSION: 2.12.2
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        submodules: true
+    - uses: ./.github/actions/install-rust
 
     # Checks for no_std support, ensure that crates can build on a no_std
     # target
@@ -523,10 +560,6 @@ jobs:
     - run: echo CARGO_BUILD_TARGET=${{ matrix.target }} >> $GITHUB_ENV
       if: matrix.target != ''
 
-    # Install OpenVINO for testing wasmtime-wasi-nn.
-    - uses: abrown/install-openvino-action@v8
-      if: runner.arch == 'X64'
-
     # Fix an ICE for now in gcc when compiling zstd with debuginfo (??)
     - run: echo CFLAGS=-g0 >> $GITHUB_ENV
       if: matrix.target == 'x86_64-pc-windows-gnu'
@@ -608,7 +641,7 @@ jobs:
         fi
 
     # Build and test all features
-    - run: ./ci/run-tests.sh ${{ matrix.extra_features }} --locked
+    - run: ./ci/run-tests.sh --locked ${{ matrix.bucket }}
       env:
         RUST_BACKTRACE: 1
 
@@ -617,6 +650,67 @@ jobs:
     # Windows fails GitHub Actions will confusingly mark the failed Windows job
     # as cancelled instead of failed.
 
+  # Test `wasmtime-wasi-nn` in its own job, as not all of its backends are
+  # compatible with all targets, and each must be tested separately anyways.
+  test_wasi_nn:
+    strategy:
+      matrix:
+        feature: ["openvino", "onnx"]
+        os: ["ubuntu-latest", "windows-latest"]
+    name: Test wasi-nn (${{ matrix.feature }}, ${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    needs: determine
+    if: needs.determine.outputs.run-full
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        submodules: true
+    - uses: ./.github/actions/install-rust
+
+    # Install OpenVINO
+    - uses: abrown/install-openvino-action@v8
+      if: runner.arch == 'X64'
+
+    # Install Rust targets.
+    - run: rustup target add wasm32-wasi
+
+    # Run the tests!
+    - run: cargo test -p wasmtime-wasi-nn --features ${{ matrix.feature }}
+      env:
+        RUST_BACKTRACE: 1
+
+    # common logic to cancel the entire run if this job fails
+    - run: gh run cancel ${{ github.run_id }}
+      if: failure() && github.event_name != 'pull_request'
+      env:
+        GH_TOKEN: ${{ github.token }}
+
+  # Test the `wasmtime-fuzzing` crate. Split out from the main tests because
+  # `--all-features` brings in OCaml, which is a pain to get setup for all
+  # targets.
+  test_fuzzing:
+    needs: determine
+    if: needs.determine.outputs.run-full
+    name: Test wasmtime-fuzzing
+    runs-on: 'ubuntu-latest'
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        submodules: true
+    - uses: ./.github/actions/install-rust
+
+    # Run the tests
+    - run: |
+        cargo test -p wasmtime-fuzzing -p wasm-spec-interpreter
+      env:
+        RUST_BACKTRACE: 1
+
+    # common logic to cancel the entire run if this job fails
+    - run: gh run cancel ${{ github.run_id }}
+      if: failure() && github.event_name != 'pull_request'
+      env:
+        GH_TOKEN: ${{ github.token }}
+
   # Test debug (DWARF) related functionality.
   test_debug_dwarf:
     needs: determine
@@ -811,6 +905,9 @@ jobs:
   # Note that `cargo nextest` is used here additionally to get parallel test
   # execution by default to help cut down on the time in CI.
   miri:
+    strategy:
+      matrix:
+        crate: ["wasmtime", "wasmtime-cli", "wasmtime-environ"]
     needs: determine
     if: needs.determine.outputs.run-full && github.repository == 'bytecodealliance/wasmtime'
     name: Miri
@@ -832,10 +929,7 @@ jobs:
     - run: echo "${{ runner.tool_cache }}/cargo-nextest/bin" >> $GITHUB_PATH
     - run: cargo install --root ${{ runner.tool_cache }}/cargo-nextest --version ${{ env.CARGO_NEXTEST_VERSION }} cargo-nextest --locked
     - run: |
-        cargo miri nextest run -j4 --no-fail-fast \
-          -p wasmtime \
-          -p wasmtime-cli \
-          -p wasmtime-environ
+        cargo miri nextest run -j4 --no-fail-fast -p ${{ matrix.crate }}
       env:
         MIRIFLAGS: -Zmiri-strict-provenance
 
@@ -924,15 +1018,18 @@ jobs:
       - test
       - test_capi
       - test_debug_dwarf
+      - test_fuzzing
+      - test_wasi_nn
+      - test_nightly
       - build
       - rustfmt
       - clangformat
       - cargo_deny
       - cargo_vet
       - doc
-      - checks
+      - micro_checks
+      - monolith_checks
       - checks_winarm64
-      - test_nightly
       - bench
       - meta_deterministic_check
       - verify-publish
diff --git a/cranelift/codegen/src/isa/riscv64/lower/isle.rs b/cranelift/codegen/src/isa/riscv64/lower/isle.rs
index f55ecd958..77299882a 100644
--- a/cranelift/codegen/src/isa/riscv64/lower/isle.rs
+++ b/cranelift/codegen/src/isa/riscv64/lower/isle.rs
@@ -18,7 +18,7 @@ use crate::machinst::{VCodeConstant, VCodeConstantData};
 use crate::{
     ir::{
         immediates::*, types::*, AtomicRmwOp, BlockCall, ExternalName, Inst, InstructionData,
-        MemFlags, Opcode, StackSlot, TrapCode, Value, ValueList,
+        MemFlags, Opcode, TrapCode, Value, ValueList,
     },
     isa::riscv64::inst::*,
     machinst::{ArgPair, InstOutput},
diff --git a/crates/fuzzing/Cargo.toml b/crates/fuzzing/Cargo.toml
index 877012436..ddaa4ad57 100644
--- a/crates/fuzzing/Cargo.toml
+++ b/crates/fuzzing/Cargo.toml
@@ -49,7 +49,7 @@ rand = { version = "0.8.0", features = ["small_rng"] }
 wasm-spec-interpreter = { path = "./wasm-spec-interpreter", optional = true, features = ['build-libinterpret'] }
 
 [features]
-fuzz-spec-interpreter = ['wasm-spec-interpreter']
+fuzz-spec-interpreter = ['dep:wasm-spec-interpreter']
 
 # Fuzz proof-carrying code. Off by default.
 fuzz-pcc = []
diff --git a/crates/jit-icache-coherence/src/libc.rs b/crates/jit-icache-coherence/src/libc.rs
index 0d7462042..9e90f1b58 100644
--- a/crates/jit-icache-coherence/src/libc.rs
+++ b/crates/jit-icache-coherence/src/libc.rs
@@ -108,6 +108,7 @@ fn riscv_flush_icache(start: u64, end: u64) -> Result<()> {
     cfg_if::cfg_if! {
         if #[cfg(feature = "one-core")] {
             use core::arch::asm;
+            let _ = (start, end);
             unsafe {
                 asm!("fence.i");
             };


Test patch:
diff --git a/ci/build-test-matrix.js b/ci/build-test-matrix.js
index 38ad67c73..4ae97b66a 100644
--- a/ci/build-test-matrix.js
+++ b/ci/build-test-matrix.js
@@ -5,43 +5,55 @@
 // couldn't figure out how to write it in bash.
 
 const fs = require('fs');
+const { spawn } = require('node:child_process');
 
-// Our first argument is a file that is a giant json blob which contains at
-// least all the messages for all of the commits that were a part of this PR.
-// This is used to test if any commit message includes a string.
-const commits = fs.readFileSync(process.argv[2]).toString();
+// Number of generic buckets to shard crates into. Note that we additionally add
+// single-crate buckets for our biggest crates.
+const GENERIC_BUCKETS = 3;
 
-// The second argument is a file that contains the names of all files modified
-// for a PR, used for file-based filters.
-const names = fs.readFileSync(process.argv[3]).toString();
+// Crates which are their own buckets. These are the very slowest to
+// compile-and-test crates.
+const SINGLE_CRATE_BUCKETS = ["wasmtime", "wasmtime-cli", "wasmtime-wasi"];
 
-// This is the full matrix of what we test on CI. This includes a number of
-// platforms and a number of cross-compiled targets that are emulated with QEMU.
-// This must be kept tightly in sync with the `test` step in `main.yml`.
+// This is the small, fast-to-execute matrix we use for PRs before they enter
+// the merge queue. Same schema as `FULL_MATRIX`.
+const FAST_MATRIX = [
+  {
+    "os": "ubuntu-latest",
+    "name": "Test Linux x86_64",
+    "filter": "linux-x64",
+    "isa": "x64",
+  },
+];
+
+// This is the full, unsharded, and unfiltered matrix of what we test on
+// CI. This includes a number of platforms and a number of cross-compiled
+// targets that are emulated with QEMU. This must be kept tightly in sync with
+// the `test` step in `main.yml`.
 //
 // The supported keys here are:
 //
 // * `os` - the github-actions name of the runner os
+//
 // * `name` - the human-readable name of the job
+//
 // * `filter` - a string which if `prtest:$filter` is in the commit messages
 //   it'll force running this test suite on PR CI.
+//
+// * `isa` - changes to `cranelift/codegen/src/$isa` will automatically run this
+//   test suite.
+//
 // * `target` - used for cross-compiles if present. Effectively Cargo's
 //   `--target` option for all its operations.
+//
 // * `gcc_package`, `gcc`, `qemu`, `qemu_target` - configuration for building
 //   QEMU and installing cross compilers to execute a cross-compiled test suite
 //   on CI.
-// * `isa` - changes to `cranelift/codegen/src/$isa` will automatically run this
-//   test suite.
+//
 // * `rust` - the Rust version to install, and if unset this'll be set to
 //   `default`
-const array = [
-  {
-    "os": "ubuntu-latest",
-    "name": "Test Linux x86_64",
-    "filter": "linux-x64",
-    "isa": "x64",
-    "extra_features": "--features wasmtime-wasi-nn/onnx"
-  },
+const FULL_MATRIX = [
+  ...FAST_MATRIX,
   {
     "os": "ubuntu-latest",
     "name": "Test MSRV on Linux x86_64",
@@ -59,20 +71,17 @@ const array = [
     "os": "macos-13",
     "name": "Test macOS x86_64",
     "filter": "macos-x64",
-    "extra_features": "--features wasmtime-wasi-nn/onnx"
   },
   {
     "os": "macos-14",
     "name": "Test macOS arm64",
     "filter": "macos-arm64",
     "target": "aarch64-apple-darwin",
-    "extra_features": "--features wasmtime-wasi-nn/onnx"
   },
   {
     "os": "windows-latest",
     "name": "Test Windows MSVC x86_64",
     "filter": "windows-x64",
-    "extra_features": "--features wasmtime-wasi-nn/onnx"
   },
   {
     "os": "windows-latest",
@@ -115,44 +124,153 @@ const array = [
   }
 ];
 
-for (let config of array) {
-  if (config.rust === undefined) {
-    config.rust = 'default';
+/// Get the workspace's full list of member crates.
+async function getWorkspaceMembers() {
+  // Spawn a `cargo metadata` subprocess, accumulate its JSON output from
+  // `stdout`, and wait for it to exit.
+  const child = spawn("cargo", ["metadata"], { encoding: "utf8" });
+  let data = "";
+  child.stdout.on("data", chunk => data += chunk);
+  await new Promise((resolve, reject) => {
+    child.on("close", resolve);
+    child.on("error", reject);
+  });
+
+  // Get the names of the crates in the workspace from the JSON metadata by
+  // building a package-id to name map and then translating the package-ids
+  // listed as workspace members.
+  const metadata = JSON.parse(data);
+  const id_to_name = {};
+  for (const pkg of metadata.packages) {
+    id_to_name[pkg.id] = pkg.name;
   }
+  return metadata.workspace_members.map(m => id_to_name[m]);
 }
 
-function myFilter(item) {
-  if (item.isa && names.includes(`cranelift/codegen/src/isa/${item.isa}`)) {
-    return true;
+/// For each given target configuration, shard the workspace's crates into
+/// buckets across that config.
+///
+/// This is essentially a `flat_map` where each config that logically tests all
+/// crates int he workspace is mapped to N sharded configs that each test only a
+/// subset of crates in the workspace. Each sharded config's subset of crates to
+/// test are disjoint from all its siblings, and the union of all thes siblings'
+/// crates to test is the full workspace members set.
+///
+/// With some poetic license around a `crates_to_test` key that doesn't actually
+/// exist, logically each element of the input `configs` list gets transformed
+/// like this:
+///
+///     { os: "ubuntu-latest", isa: "x64", ..., crates: "all" }
+///
+///     ==>
+///
+///     [
+///       { os: "ubuntu-latest", isa: "x64", ..., crates: ["wasmtime"] },
+///       { os: "ubuntu-latest", isa: "x64", ..., crates: ["wasmtime-cli"] },
+///       { os: "ubuntu-latest", isa: "x64", ..., crates: ["wasmtime-wasi"] },
+///       { os: "ubuntu-latest", isa: "x64", ..., crates: ["cranelift", "cranelift-codegen", ...] },
+///       { os: "ubuntu-latest", isa: "x64", ..., crates: ["wasmtime-slab", "cranelift-entity", ...] },
+///       { os: "ubuntu-latest", isa: "x64", ..., crates: ["cranelift-environ", "wasmtime-cli-flags", ...] },
+///       ...
+///     ]
+///
+/// Note that `crates: "all"` is implicit in the input and omitted. Similarly,
+/// `crates: [...]` in each output config is actually implemented via adding a
+/// `bucket` key, which contains the CLI flags we must pass to `cargo` to run
+/// tests for just this config's subset of crates.
+async function shard(configs) {
+  const members = await getWorkspaceMembers();
+
+  // Divide the workspace crates into N disjoint subsets. Crates that are
+  // particularly expensive to compile and test form their own singleton subset.
+  const buckets = Array.from({ length: GENERIC_BUCKETS }, _ => new Set());
+  let i = 0;
+  for (const crate of members) {
+    if (SINGLE_CRATE_BUCKETS.indexOf(crate) != -1) continue;
+    buckets[i].add(crate);
+    i = (i + 1) % GENERIC_BUCKETS;
   }
-  if (item.filter && commits.includes(`prtest:${item.filter}`)) {
-    return true;
+  for (crate of SINGLE_CRATE_BUCKETS) {
+    buckets.push(new Set([crate]));
   }
 
-  // If any runtest was modified, re-run the whole test suite as those can
-  // target any backend.
-  if (names.includes(`cranelift/filetests/filetests/runtests`)) {
-    return true;
+  // For each config, expand it into N configs, one for each disjoint set we
+  // created above.
+  const sharded = [];
+  for (const config of configs) {
+    for (const bucket of buckets) {
+      sharded.push(Object.assign(
+        {},
+        config,
+        {
+          name: `${config.name} (${Array.from(bucket).join(', ')})`,
+          // We run tests via `cargo test --workspace`, so exclude crates that
+          // aren't in this bucket, rather than naming only the crates that are
+          // in this bucket.
+          bucket: members
+            .map(c => bucket.has(c) ? `--package ${c}` : `--exclude ${c}`)
+            .join(" "),
+        }
+      ));
+    }
   }
-
-  return false;
+  return sharded;
 }
 
-const filtered = array.filter(myFilter);
+async function main() {
+  // Our first argument is a file that is a giant json blob which contains at
+  // least all the messages for all of the commits that were a part of this PR.
+  // This is used to test if any commit message includes a string.
+  const commits = fs.readFileSync(process.argv[2]).toString();
 
-// If the optional third argument to this script is `true` then that means all
-// tests are being run and no filtering should happen.
-if (process.argv[4] == 'true') {
-  console.log(JSON.stringify(array));
-  return;
-}
+  // The second argument is a file that contains the names of all files modified
+  // for a PR, used for file-based filters.
+  const names = fs.readFileSync(process.argv[3]).toString();
+
+  for (let config of FULL_MATRIX) {
+    if (config.rust === undefined) {
+      config.rust = 'default';
+    }
+  }
+
+  // If the optional third argument to this script is `true` then that means all
+  // tests are being run and no filtering should happen.
+  if (process.argv[4] == 'true') {
+    console.log(JSON.stringify(await shard(FULL_MATRIX), undefined, 2));
+    return;
+  }
+
+  // When we aren't running the full CI matrix, filter configs down to just the
+  // relevant bits based on files changed in this commit or if the commit asks
+  // for a certain config to run.
+  const filtered = FULL_MATRIX.filter(config => {
+    // If an ISA-specific test was modified, then include that ISA config.
+    if (config.isa && names.includes(`cranelift/codegen/src/isa/${config.isa}`)) {
+      return true;
+    }
+
+    // If any runtest was modified, include all ISA configs as runtests can
+    // target any backend.
+    if (names.includes(`cranelift/filetests/filetests/runtests`)) {
+      return config.isa !== undefined;
+    }
+
+    // If the commit explicitly asks for this test config, then include it.
+    if (config.filter && commits.includes(`prtest:${config.filter}`)) {
+      return true;
+    }
+
+    return false;
+  });
+
+  // If at least one test is being run via our filters then run those tests.
+  if (filtered.length > 0) {
+    console.log(JSON.stringify(await shard(filtered), undefined, 2));
+    return;
+  }
 
-// If at least one test is being run via our filters then run those tests.
-if (filtered.length > 0) {
-  console.log(JSON.stringify(filtered));
-  return;
+  // Otherwise if nothing else is being run, run the fast subset of the matrix.
+  console.log(JSON.stringify(await shard(FAST_MATRIX), undefined, 2));
 }
 
-// Otherwise if nothing else is being run, run the first one which is Ubuntu
-// Linux which should be the fastest for now.
-console.log(JSON.stringify([array[0]]));
+main()
diff --git a/ci/run-tests.sh b/ci/run-tests.sh
index db338dc24..274d35232 100755
--- a/ci/run-tests.sh
+++ b/ci/run-tests.sh
@@ -1,10 +1,23 @@
-#!/bin/bash
+#!/usr/bin/env bash
+
+# Excludes:
+#
+# - test-programs: just programs used in tests.
+#
+# - wasmtime-wasi-nn: mutually-exclusive features that aren't available for all
+#   targets, needs its own CI job.
+#
+# - wasmtime-fuzzing: enabling all features brings in OCaml which is a pain to
+#   configure for all targets, so it has its own CI job.
+#
+# - wasm-spec-interpreter: brings in OCaml which is a pain to configure for all
+#   targets, tested as part of the wastime-fuzzing CI job.
 
 cargo test \
-    --features wasi-threads \
-    --features wasi-http \
-    --features component-model \
-    --features serve \
-    --workspace \
-    --exclude test-programs \
-    $@
+      --workspace \
+      --all-features \
+      --exclude test-programs \
+      --exclude wasmtime-wasi-nn \
+      --exclude wasmtime-fuzzing \
+      --exclude wasm-spec-interpreter \
+      $@
diff --git a/tests/all/pooling_allocator.rs b/tests/all/pooling_allocator.rs
index eceb330fc..7c563a0e1 100644
--- a/tests/all/pooling_allocator.rs
+++ b/tests/all/pooling_allocator.rs
@@ -644,10 +644,10 @@ fn instance_too_large() -> Result<()> {
     let engine = Engine::new(&config)?;
     let expected = if cfg!(feature = "wmemcheck") {
         "\
-        instance allocation for this module requires 336 bytes which exceeds the \
+        instance allocation for this module requires 352 bytes which exceeds the \
 configured maximum of 16 bytes; breakdown of allocation requirement:
 
- * 76.19% - 256 bytes - instance state management
+ * 72.73% - 256 bytes - instance state management
 "
     } else {
         "\
@@ -671,11 +671,11 @@ configured maximum of 16 bytes; breakdown of allocation requirement:
 
     let expected = if cfg!(feature = "wmemcheck") {
         "\
-instance allocation for this module requires 1936 bytes which exceeds the \
+instance allocation for this module requires 1952 bytes which exceeds the \
 configured maximum of 16 bytes; breakdown of allocation requirement:
 
- * 13.22% - 256 bytes - instance state management
- * 82.64% - 1600 bytes - defined globals
+ * 13.11% - 256 bytes - instance state management
+ * 81.97% - 1600 bytes - defined globals
 "
     } else {
         "\