Repository: metal3-io/baremetal-operator

Issue:
bmo-optional-pull fails consistently at docker issues
### Which jobs are failing?

BMO optional pull.  Note that BMO optional workflow is not, so it looks like setup issue in Prow version.

### Which tests are failing?

```
[2025-12-08T07:38:16.856Z]   [FAILED] Unexpected error:

[2025-12-08T07:38:16.856Z]       <*errors.withStack | 0xc0008280a8>: 

[2025-12-08T07:38:16.856Z]       Failed to load image "quay.io/metal3-io/baremetal-operator:release-0.9" into the kind cluster "bmo-e2e": failed to load image: command "docker exec --privileged -i bmo-e2e-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1
```

### Since when has it been failing?

Since last week at minimum, probably long but it hasn't been used really.

### Jenkins link

https://jenkins.nordix.org/view/Metal3/job/metal3-bmo-e2e-test-optional-pull/58/

### Reason for failure (if possible)

_No response_

### Anything else we need to know?

_No response_

### Label(s) to be applied

/kind failing-test
One or more /area label. See https://github.com/metal3-io/baremetal-operator/labels for the list of labels.


Discussion:
@tuminoid: /triage accepted

Bot analysis outcome (to be human verified):

```
   === BUILD DEBUG: metal3-bmo-e2e-test-optional-pull #58 ===

   Context:

     - Health: 93.8% failure rate, 15 consecutive failures
     - Started failing: build #54 (Dec 5, 2025)
     - Last success: build #38 (Nov 4, 2025)
     - Build #53 (Nov 12) used CAPI v1.12.0-beta.0 and image loading worked

   Root Cause: CAPI v1.12.0-rc.0 + containerd image import incompatibility

   The failure occurs when loading images into kind cluster during test setup. The error:

     Failed to load image "quay.io/metal3-io/baremetal-operator:release-0.9" into the kind cluster "bmo-e2e": 
     failed to load image: command "docker exec --privileged -i bmo-e2e-control-plane ctr --namespace=k8s.io 
     images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1

   Technical Details:

     - PR #2816 merged on Dec 1, 2025 upgraded CAPI from v1.12.0-beta.0 to v1.12.0-rc.0
     - Kind uses ctr images import --all-platforms which expects all platform layers to be present
     - When Docker pulls a single-arch image (like quay.io/metal3-io/baremetal-operator:release-0.9), it only fetches layers for the current platform
     - Containerd's ctr import --all-platforms then fails because referenced platform-specific blobs are missing
     - This is known kind issue #3795 and #3996 (marked as duplicate)

   Evidence:

     - Build #38 (Nov 4): SUCCESS - used CAPI v1.11.2
     - Build #53 (Nov 12): Image loading worked (CAPI v1.12.0-beta.0), failed on actual test
     - Build #54 (Dec 5): Image loading FAILS (CAPI v1.12.0-rc.0)
     - Same error in builds #54-58 consistently
     - quay.io/metal3-io/baremetal-operator:release-0.9 is NOT a manifest list (is_manifest_list=False)

   Recommendations:

     - Immediate workaround: Pin CAPI to v1.12.0-beta.0 or v1.11.2 until fix is available
     - Better fix: Build BMO release images as multi-arch manifest lists
     - Alternative: Modify e2e test to pre-pull images with docker pull before loading (so they're cached locally)
     - Track: Monitor kind issue #3795 for upstream fix
```

Key seems to be specifically this 0.9 image, which is going away soon anyways.@tuminoid: Issues mentioned:
- https://github.com/kubernetes-sigs/kind/issues/3795 @tuminoid: CAPD hitting the same: https://github.com/kubernetes-sigs/cluster-api/pull/13113@tuminoid: /assign

Solution patch:
diff --git a/hack/tools/go.mod b/hack/tools/go.mod
index 43972656..c6d407ee 100644
--- a/hack/tools/go.mod
+++ b/hack/tools/go.mod
@@ -7,7 +7,7 @@ require (
 	github.com/google/go-github v17.0.0+incompatible
 	golang.org/x/crypto v0.45.0
 	golang.org/x/oauth2 v0.33.0
-	sigs.k8s.io/cluster-api/test v1.12.0-rc.0
+	sigs.k8s.io/cluster-api/test v1.12.0-rc.1
 	sigs.k8s.io/controller-tools v0.16.5
 	sigs.k8s.io/kustomize/api v0.21.0
 	sigs.k8s.io/kustomize/kustomize/v5 v5.8.0
diff --git a/hack/tools/go.sum b/hack/tools/go.sum
index 40530ba2..57ee211b 100644
--- a/hack/tools/go.sum
+++ b/hack/tools/go.sum
@@ -191,8 +191,8 @@ k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOP
 k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts=
 k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y=
 k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
-sigs.k8s.io/cluster-api/test v1.12.0-rc.0 h1:xQuRM5nVY2dL8USqXRDnt4kT1MChK93BvpbQwonnNhc=
-sigs.k8s.io/cluster-api/test v1.12.0-rc.0/go.mod h1:eS05aBCEZeKzcW7MGnT9U+zap4NCpkxHBWb10DshWp0=
+sigs.k8s.io/cluster-api/test v1.12.0-rc.1 h1:v/hXhN0mnIj8Z6gNk2Okt/XQycfRgvxDNNRcipS1xDA=
+sigs.k8s.io/cluster-api/test v1.12.0-rc.1/go.mod h1:eS05aBCEZeKzcW7MGnT9U+zap4NCpkxHBWb10DshWp0=
 sigs.k8s.io/controller-tools v0.16.5 h1:5k9FNRqziBPwqr17AMEPPV/En39ZBplLAdOwwQHruP4=
 sigs.k8s.io/controller-tools v0.16.5/go.mod h1:8vztuRVzs8IuuJqKqbXCSlXcw+lkAv/M2sTpg55qjMY=
 sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=



Test patch:
diff --git a/test/e2e/e2e_config.go b/test/e2e/e2e_config.go
index 4687bd94..ba5e5f72 100644
--- a/test/e2e/e2e_config.go
+++ b/test/e2e/e2e_config.go
@@ -33,6 +33,18 @@ const (
 	ironicString string = "ironic"
 )
 
+// ContainerImage describes an image to load into a cluster.
+// This is a copy of clusterctl.ContainerImage with proper yaml tags for YAML unmarshaling.
+// See: https://github.com/metal3-io/baremetal-operator/issues/2832
+type ContainerImage struct {
+	// Name is the fully qualified name of the image.
+	Name string `yaml:"name"`
+
+	// LoadBehavior may be used to dictate whether a failed load operation
+	// should fail the test run.
+	LoadBehavior clusterctl.LoadImageBehavior `yaml:"loadBehavior,omitempty"`
+}
+
 type BMOIronicUpgradeInput struct {
 	// DeployIronic determines if Ironic should be installed at the beginning of the test.
 	// This should be generally set to `true`, but can be `false` in case Ironic is either pre-installed
@@ -57,13 +69,14 @@ type BMOIronicUpgradeInput struct {
 type Config struct {
 	// Images is a list of container images to load into the Kind cluster.
 	// Note that this not relevant when using an existing cluster.
-	Images []clusterctl.ContainerImage `json:"images,omitempty"`
+	// Uses custom ContainerImage type with yaml tags for proper YAML unmarshaling.
+	Images []ContainerImage `yaml:"images,omitempty"`
 
 	// Variables to be used in the tests.
-	Variables map[string]string `json:"variables,omitempty"`
+	Variables map[string]string `yaml:"variables,omitempty"`
 
 	// Intervals to be used for long operations during tests.
-	Intervals map[string][]string `json:"intervals,omitempty"`
+	Intervals map[string][]string `yaml:"intervals,omitempty"`
 
 	// BMOIronicUpgradeSpecs
 	BMOIronicUpgradeSpecs []BMOIronicUpgradeInput `yaml:"bmoIronicUpgradeSpecs,omitempty"`
@@ -100,6 +113,19 @@ func (c *Config) Defaults() {
 	}
 }
 
+// GetClusterctlImages converts the local ContainerImage slice to clusterctl.ContainerImage
+// for use with the CAPI bootstrap framework.
+func (c *Config) GetClusterctlImages() []clusterctl.ContainerImage {
+	result := make([]clusterctl.ContainerImage, len(c.Images))
+	for i, img := range c.Images {
+		result[i] = clusterctl.ContainerImage{
+			Name:         img.Name,
+			LoadBehavior: img.LoadBehavior,
+		}
+	}
+	return result
+}
+
 // Validate validates the configuration. More specifically:
 // - Image should have name and loadBehavior be one of [mustload, tryload].
 // - Intervals should be valid ginkgo intervals.
diff --git a/test/e2e/e2e_suite_test.go b/test/e2e/e2e_suite_test.go
index ca17f9a4..e376e647 100644
--- a/test/e2e/e2e_suite_test.go
+++ b/test/e2e/e2e_suite_test.go
@@ -88,7 +88,7 @@ var _ = SynchronizedBeforeSuite(func() []byte {
 	} else {
 		clusterProvider = bootstrap.CreateKindBootstrapClusterAndLoadImages(ctx, bootstrap.CreateKindBootstrapClusterAndLoadImagesInput{
 			Name:              "bmo-e2e",
-			Images:            e2eConfig.Images,
+			Images:            e2eConfig.GetClusterctlImages(),
 			ExtraPortMappings: e2eConfig.KindExtraPortMappings,
 		})
 		Expect(clusterProvider).ToNot(BeNil(), "Failed to create a cluster")
diff --git a/test/e2e/upgrade_test.go b/test/e2e/upgrade_test.go
index b04c86db..712b532d 100644
--- a/test/e2e/upgrade_test.go
+++ b/test/e2e/upgrade_test.go
@@ -365,7 +365,7 @@ var _ = Describe("Upgrade", Label("optional", "upgrade"), func() {
 			upgradeClusterName := fmt.Sprintf("bmo-e2e-upgrade-%d", GinkgoParallelProcess())
 			upgradeClusterProvider = bootstrap.CreateKindBootstrapClusterAndLoadImages(ctx, bootstrap.CreateKindBootstrapClusterAndLoadImagesInput{
 				Name:              upgradeClusterName,
-				Images:            e2eConfig.Images,
+				Images:            e2eConfig.GetClusterctlImages(),
 				ExtraPortMappings: e2eConfig.KindExtraPortMappings,
 			})
 			Expect(upgradeClusterProvider).ToNot(BeNil(), "Failed to create a cluster")
diff --git a/test/go.mod b/test/go.mod
index 45188e25..c09e1d58 100644
--- a/test/go.mod
+++ b/test/go.mod
@@ -17,8 +17,8 @@ require (
 	k8s.io/klog/v2 v2.130.1
 	k8s.io/utils v0.0.0-20250604170112-4c0f3b243397
 	libvirt.org/go/libvirt v1.11010.0
-	sigs.k8s.io/cluster-api v1.12.0-rc.0
-	sigs.k8s.io/cluster-api/test v1.12.0-rc.0
+	sigs.k8s.io/cluster-api v1.12.0-rc.1
+	sigs.k8s.io/cluster-api/test v1.12.0-rc.1
 	sigs.k8s.io/controller-runtime v0.22.4
 	sigs.k8s.io/kind v0.30.0
 	sigs.k8s.io/kustomize/api v0.21.0
diff --git a/test/go.sum b/test/go.sum
index 40760140..97d2c4f6 100644
--- a/test/go.sum
+++ b/test/go.sum
@@ -415,10 +415,10 @@ libvirt.org/go/libvirt v1.11010.0 h1:1EIh2x6qcRoIBBOvrgN62vq5FIpgUBrmGadprQ/4M0Y
 libvirt.org/go/libvirt v1.11010.0/go.mod h1:1WiFE8EjZfq+FCVog+rvr1yatKbKZ9FaFMZgEqxEJqQ=
 sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 h1:jpcvIRr3GLoUoEKRkHKSmGjxb6lWwrBlJsXc+eUYQHM=
 sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2/go.mod h1:Ve9uj1L+deCXFrPOk1LpFXqTg7LCFzFso6PA48q/XZw=
-sigs.k8s.io/cluster-api v1.12.0-rc.0 h1:7tWLgsGF/5zd0tdQloxdpBG1eEkYfgIFWFdYZaLVLF0=
-sigs.k8s.io/cluster-api v1.12.0-rc.0/go.mod h1:+S6WJdi8UPdqv5q9nka5al3ed/Qa0zAcSBgzTaa9VKA=
-sigs.k8s.io/cluster-api/test v1.12.0-rc.0 h1:xQuRM5nVY2dL8USqXRDnt4kT1MChK93BvpbQwonnNhc=
-sigs.k8s.io/cluster-api/test v1.12.0-rc.0/go.mod h1:eS05aBCEZeKzcW7MGnT9U+zap4NCpkxHBWb10DshWp0=
+sigs.k8s.io/cluster-api v1.12.0-rc.1 h1:ZoZtg/kRU4av5qt1z3MhbBP6hS1SPfWguFsbOiMX/Uc=
+sigs.k8s.io/cluster-api v1.12.0-rc.1/go.mod h1:+S6WJdi8UPdqv5q9nka5al3ed/Qa0zAcSBgzTaa9VKA=
+sigs.k8s.io/cluster-api/test v1.12.0-rc.1 h1:v/hXhN0mnIj8Z6gNk2Okt/XQycfRgvxDNNRcipS1xDA=
+sigs.k8s.io/cluster-api/test v1.12.0-rc.1/go.mod h1:eS05aBCEZeKzcW7MGnT9U+zap4NCpkxHBWb10DshWp0=
 sigs.k8s.io/controller-runtime v0.22.4 h1:GEjV7KV3TY8e+tJ2LCTxUTanW4z/FmNB7l327UfMq9A=
 sigs.k8s.io/controller-runtime v0.22.4/go.mod h1:+QX1XUpTXN4mLoblf4tqr5CQcyHPAki2HLXqQMY6vh8=
 sigs.k8s.io/gateway-api v1.1.0 h1:DsLDXCi6jR+Xz8/xd0Z1PYl2Pn0TyaFMOPPZIj4inDM=