Repository: moby/buildkit

Issue:
WCOW integration tests should be split up more in CI
With more integration tests being enabled for WCOW they have become the bottleneck of getting CI passing, taking almost an hour to complete for the longest jobs. We need to split up these suites more so they run in parallel and don't take more time than the Linux tests. They also seem to be more flaky than Linux tests, but that can be looked later after split.

@profnandaa @billywr

Discussion:
@crazy-max: Seems the one that takes almost an hour is `./frontend/dockerfile`: https://github.com/moby/buildkit/actions/runs/13614348309/job/38055570722#step:8:3

![Image](https://github.com/user-attachments/assets/0af2c6b2-39a5-4deb-ae00-c12c71c1188b)

Linux one is just about 4min: https://github.com/moby/buildkit/actions/runs/13583450488/job/37973505089#step:8:37

Not sure why we have such difference. I tried to enable parallelism in https://github.com/moby/buildkit/pull/5801 but doesn't seem to do the trick.

Comparing a test on Windows:

```
--- PASS: TestIntegration/TestDockerfileFromHTTP/worker=containerd/frontend=builtin (16.65s)
--- PASS: TestIntegration/TestDockerfileFromHTTP/worker=containerd/frontend=client (18.17s)
```

With one on Linux:


```
--- PASS: TestIntegration/TestDockerfileFromHTTP/worker=containerd/frontend=client (1.08s)
--- PASS: TestIntegration/TestDockerfileFromHTTP/worker=containerd/frontend=builtin (1.16s)
```

The diff is quite huge. Could it be the base image https://github.com/moby/buildkit/blob/824bc35b9771357e548ca913f8790854b70cf1f5/frontend/dockerfile/dockerfile_test.go#L4677

When trying to do a local export? https://github.com/moby/buildkit/blob/824bc35b9771357e548ca913f8790854b70cf1f5/frontend/dockerfile/dockerfile_test.go#L4705-L4710@crazy-max: Guess we need https://github.com/moby/buildkit/issues/5264@profnandaa: let me take a look at this starting tomorrow.

> Guess we need [#5264](https://github.com/moby/buildkit/issues/5264)

@crazy-max -- I'm not entirely sure this will help speed up the CI since "scratch-based" tests are only a portion. Might you have any other ideas on how we can plit the tests?@crazy-max: > Might you have any other ideas on how we can plit the tests?

We could distribute tests using a matrix with https://github.com/crazy-max/gotestlist?tab=readme-ov-file#distribute-tests-with-github-actions. Used on moby repo: https://github.com/moby/moby/blob/41f781fab3cae181cc9be3ec93cd91b99466fa84/.github/workflows/.windows.yml#L252-L262@profnandaa: Thanks for this, will take a look.@tonistiigi: I don't think gotestlist would work with integration pkg, but we could add some slicing directly to the integration test runner.@profnandaa: fixed in #5806 

Solution patch:
diff --git a/.github/workflows/test-os.yml b/.github/workflows/test-os.yml
index 42e92c8b2..5ba92ba12 100644
--- a/.github/workflows/test-os.yml
+++ b/.github/workflows/test-os.yml
@@ -89,7 +89,18 @@ jobs:
           - ./worker/containerd
           - ./solver
           - ./frontend
-          - ./frontend/dockerfile
+          - ./frontend/dockerfile#1-12
+          - ./frontend/dockerfile#2-12
+          - ./frontend/dockerfile#3-12
+          - ./frontend/dockerfile#4-12
+          - ./frontend/dockerfile#5-12
+          - ./frontend/dockerfile#6-12
+          - ./frontend/dockerfile#7-12
+          - ./frontend/dockerfile#8-12
+          - ./frontend/dockerfile#9-12
+          - ./frontend/dockerfile#10-12
+          - ./frontend/dockerfile#11-12
+          - ./frontend/dockerfile#12-12
         include:
           - os: windows-2022
             worker: containerd
@@ -103,10 +114,18 @@ jobs:
       -
         name: Prepare
         run: |
+          TESTPKG=$(echo "${{ matrix.pkg }}" | awk '-F#' '{print $1}')
+          TESTSLICE=$(echo "${{ matrix.pkg }}" | awk '-F#' '{print $2}')
+          echo "TESTPKG=$TESTPKG" >> $GITHUB_ENV
           echo "TEST_REPORT_NAME=${{ github.job }}-$(echo "${{ matrix.pkg }}-${{ matrix.skip-integration-tests }}-${{ matrix.worker }}" | tr -dc '[:alnum:]-\n\r' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV
           testFlags="${{ env.TESTFLAGS }}"
+          testSliceOffset=""
+          if [ -n "$TESTSLICE" ]; then
+            echo "TESTSLICE=$TESTSLICE" >> $GITHUB_ENV
+            testSliceOffset="slice=$TESTSLICE/"
+          fi
           if [ -n "${{ matrix.worker }}" ]; then
-            testFlags="${testFlags} --run=TestIntegration/.*/worker=${{ matrix.worker }}"
+            testFlags="${testFlags} --run=TestIntegration/$testSliceOffset.*/worker=${{ matrix.worker }}"
           fi
           echo "TESTFLAGS=${testFlags}" >> $GITHUB_ENV
         shell: bash
@@ -142,7 +161,7 @@ jobs:
           gotestsum \
             --jsonfile="./bin/testreports/go-test-report-${{ env.TEST_REPORT_NAME }}.json" \
             --junitfile="./bin/testreports/junit-report-${{ env.TEST_REPORT_NAME }}.xml" \
-            --packages="${{ matrix.pkg }}" \
+            --packages="${{ env.TESTPKG }}" \
             -- \
               "-mod=vendor" \
               "-coverprofile" "./bin/testreports/coverage-${{ env.TEST_REPORT_NAME }}.txt" \



Test patch:
diff --git a/cache/remotecache/gha/gha_test.go b/cache/remotecache/gha/gha_test.go
index 6da0e85da..6b80d18f5 100644
--- a/cache/remotecache/gha/gha_test.go
+++ b/cache/remotecache/gha/gha_test.go
@@ -127,6 +127,10 @@ func testBasicGhaCacheImportExportExtraTimeout(t *testing.T, sb integration.Sand
 	}
 	maps.Copy(cacheImportAttrs, cacheAttrs)
 
+	// Github Cache Service v2 has a problem of not being immediately consistent
+	// so we need to wait for the changes to propagate.
+	time.Sleep(3 * time.Second)
+
 	_, err = c.Solve(sb.Context(), def, client.SolveOpt{
 		Exports: []client.ExportEntry{
 			{
diff --git a/util/testutil/integration/run.go b/util/testutil/integration/run.go
index 2c0810291..7dd1e2262 100644
--- a/util/testutil/integration/run.go
+++ b/util/testutil/integration/run.go
@@ -5,6 +5,7 @@ import (
 	"context"
 	"fmt"
 	"maps"
+	"math"
 	"math/rand"
 	"os"
 	"os/exec"
@@ -12,6 +13,7 @@ import (
 	"reflect"
 	"runtime"
 	"sort"
+	"strconv"
 	"strings"
 	"sync"
 	"testing"
@@ -161,6 +163,24 @@ func Run(t *testing.T, testCases []Test, opt ...TestOpt) {
 		t.Skip("skipping integration tests")
 	}
 
+	var sliceSplit int
+	v, ok := os.LookupEnv("TESTSLICE")
+	if ok {
+		t.Logf("TESTSLICE=%s", v)
+		offsetS, totalS, ok := strings.Cut(v, "-")
+		if !ok {
+			t.Fatalf("invalid TESTSLICE=%s", v)
+		}
+		offset, err := strconv.Atoi(offsetS)
+		require.NoError(t, err)
+		total, err := strconv.Atoi(totalS)
+		require.NoError(t, err)
+		if offset < 1 || total < 1 || offset > total {
+			t.Fatalf("invalid TESTSLICE=%s", v)
+		}
+		sliceSplit = total
+	}
+
 	var tc testConf
 	for _, o := range opt {
 		o(&tc)
@@ -182,9 +202,14 @@ func Run(t *testing.T, testCases []Test, opt ...TestOpt) {
 	})
 
 	for _, br := range list {
-		for _, tc := range testCases {
+		for i, tc := range testCases {
 			for _, mv := range matrix {
 				fn := tc.Name()
+				if sliceSplit > 0 {
+					pageLimit := int(math.Ceil(float64(len(testCases)) / float64(sliceSplit)))
+					sliceName := fmt.Sprintf("slice=%d-%d/", i/pageLimit+1, sliceSplit)
+					fn = sliceName + fn
+				}
 				name := fn + "/worker=" + br.Name() + mv.functionSuffix()
 				func(fn, testName string, br Worker, tc Test, mv matrixValue) {
 					ok := t.Run(testName, func(t *testing.T) {