Repository: metal3-io/baremetal-operator

Issue:
bmo-optional-pull fails consistently at docker issues
### Which jobs are failing?

BMO optional pull.  Note that BMO optional workflow is not, so it looks like setup issue in Prow version.

### Which tests are failing?

```
[2025-12-08T07:38:16.856Z]   [FAILED] Unexpected error:

[2025-12-08T07:38:16.856Z]       <*errors.withStack | 0xc0008280a8>: 

[2025-12-08T07:38:16.856Z]       Failed to load image "quay.io/metal3-io/baremetal-operator:release-0.9" into the kind cluster "bmo-e2e": failed to load image: command "docker exec --privileged -i bmo-e2e-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1
```

### Since when has it been failing?

Since last week at minimum, probably long but it hasn't been used really.

### Jenkins link

https://jenkins.nordix.org/view/Metal3/job/metal3-bmo-e2e-test-optional-pull/58/

### Reason for failure (if possible)

_No response_

### Anything else we need to know?

_No response_

### Label(s) to be applied

/kind failing-test
One or more /area label. See https://github.com/metal3-io/baremetal-operator/labels for the list of labels.


Discussion:
@tuminoid: /triage accepted

Bot analysis outcome (to be human verified):

```
   === BUILD DEBUG: metal3-bmo-e2e-test-optional-pull #58 ===

   Context:

     - Health: 93.8% failure rate, 15 consecutive failures
     - Started failing: build #54 (Dec 5, 2025)
     - Last success: build #38 (Nov 4, 2025)
     - Build #53 (Nov 12) used CAPI v1.12.0-beta.0 and image loading worked

   Root Cause: CAPI v1.12.0-rc.0 + containerd image import incompatibility

   The failure occurs when loading images into kind cluster during test setup. The error:

     Failed to load image "quay.io/metal3-io/baremetal-operator:release-0.9" into the kind cluster "bmo-e2e": 
     failed to load image: command "docker exec --privileged -i bmo-e2e-control-plane ctr --namespace=k8s.io 
     images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1

   Technical Details:

     - PR #2816 merged on Dec 1, 2025 upgraded CAPI from v1.12.0-beta.0 to v1.12.0-rc.0
     - Kind uses ctr images import --all-platforms which expects all platform layers to be present
     - When Docker pulls a single-arch image (like quay.io/metal3-io/baremetal-operator:release-0.9), it only fetches layers for the current platform
     - Containerd's ctr import --all-platforms then fails because referenced platform-specific blobs are missing
     - This is known kind issue #3795 and #3996 (marked as duplicate)

   Evidence:

     - Build #38 (Nov 4): SUCCESS - used CAPI v1.11.2
     - Build #53 (Nov 12): Image loading worked (CAPI v1.12.0-beta.0), failed on actual test
     - Build #54 (Dec 5): Image loading FAILS (CAPI v1.12.0-rc.0)
     - Same error in builds #54-58 consistently
     - quay.io/metal3-io/baremetal-operator:release-0.9 is NOT a manifest list (is_manifest_list=False)

   Recommendations:

     - Immediate workaround: Pin CAPI to v1.12.0-beta.0 or v1.11.2 until fix is available
     - Better fix: Build BMO release images as multi-arch manifest lists
     - Alternative: Modify e2e test to pre-pull images with docker pull before loading (so they're cached locally)
     - Track: Monitor kind issue #3795 for upstream fix
```

Key seems to be specifically this 0.9 image, which is going away soon anyways.@tuminoid: Issues mentioned:
- https://github.com/kubernetes-sigs/kind/issues/3795 @tuminoid: CAPD hitting the same: https://github.com/kubernetes-sigs/cluster-api/pull/13113@tuminoid: /assign

Solution patch:
diff --git a/.markdownlint-cli2.yaml b/.markdownlint-cli2.yaml
index 7811fc66..944fa503 100644
--- a/.markdownlint-cli2.yaml
+++ b/.markdownlint-cli2.yaml
@@ -4,6 +4,9 @@ config:
   ul-indent:
     # Kramdown wanted us to have 3 earlier, tho this CLI recommends 2 or 4
     indent: 3
+  line-length:
+    tables: false
+    code_blocks: false
 
 # Don't autofix anything, we're linting here
 fix: false
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 355d31d9..be41cb9c 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -138,15 +138,11 @@ the ability to backport and release patch versions;
 - The EOL date of each API Version is determined from the last release available
   once a new API version is published.
 
-<!-- markdownlint-disable MD013 -->
-
 | API Version  | Maintained Until                                              |
 | ------------ | ------------------------------------------------------------- |
 | **v1alpha1** | TBD (current latest)                                          |
 | **v1beta1**  | Upcoming (Proposal [PR](https://github.com/metal3-io/metal3-docs/pull/332)) |
 
-<!-- markdownlint-enable MD013 -->
-
 - For the current stable API version (v1alpha1) we support the two most recent
   minor releases; older minor releases are immediately unsupported when a new
   major/minor release is available.
diff --git a/hack/tools/go.mod b/hack/tools/go.mod
index 43972656..c6d407ee 100644
--- a/hack/tools/go.mod
+++ b/hack/tools/go.mod
@@ -7,7 +7,7 @@ require (
 	github.com/google/go-github v17.0.0+incompatible
 	golang.org/x/crypto v0.45.0
 	golang.org/x/oauth2 v0.33.0
-	sigs.k8s.io/cluster-api/test v1.12.0-rc.0
+	sigs.k8s.io/cluster-api/test v1.12.0-rc.1
 	sigs.k8s.io/controller-tools v0.16.5
 	sigs.k8s.io/kustomize/api v0.21.0
 	sigs.k8s.io/kustomize/kustomize/v5 v5.8.0
diff --git a/hack/tools/go.sum b/hack/tools/go.sum
index 40530ba2..57ee211b 100644
--- a/hack/tools/go.sum
+++ b/hack/tools/go.sum
@@ -191,8 +191,8 @@ k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOP
 k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts=
 k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y=
 k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
-sigs.k8s.io/cluster-api/test v1.12.0-rc.0 h1:xQuRM5nVY2dL8USqXRDnt4kT1MChK93BvpbQwonnNhc=
-sigs.k8s.io/cluster-api/test v1.12.0-rc.0/go.mod h1:eS05aBCEZeKzcW7MGnT9U+zap4NCpkxHBWb10DshWp0=
+sigs.k8s.io/cluster-api/test v1.12.0-rc.1 h1:v/hXhN0mnIj8Z6gNk2Okt/XQycfRgvxDNNRcipS1xDA=
+sigs.k8s.io/cluster-api/test v1.12.0-rc.1/go.mod h1:eS05aBCEZeKzcW7MGnT9U+zap4NCpkxHBWb10DshWp0=
 sigs.k8s.io/controller-tools v0.16.5 h1:5k9FNRqziBPwqr17AMEPPV/En39ZBplLAdOwwQHruP4=
 sigs.k8s.io/controller-tools v0.16.5/go.mod h1:8vztuRVzs8IuuJqKqbXCSlXcw+lkAv/M2sTpg55qjMY=
 sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=
diff --git a/internal/controller/metal3.io/baremetalhost_controller.go b/internal/controller/metal3.io/baremetalhost_controller.go
index 51941c88..51d80b4f 100644
--- a/internal/controller/metal3.io/baremetalhost_controller.go
+++ b/internal/controller/metal3.io/baremetalhost_controller.go
@@ -515,7 +515,8 @@ func (r *BareMetalHostReconciler) actionPowerOffBeforeDeleting(prov provisioner.
 	info.log.Info("host ready to be powered off")
 	provResult, err := prov.PowerOff(
 		metal3api.RebootModeHard,
-		info.host.Status.ErrorType == metal3api.PowerManagementError)
+		info.host.Status.ErrorType == metal3api.PowerManagementError,
+		info.host.Spec.AutomatedCleaningMode)
 
 	if err != nil {
 		return actionError{fmt.Errorf("failed to power off before deleting node: %w", err)}
@@ -1639,7 +1640,7 @@ func (r *BareMetalHostReconciler) manageHostPower(prov provisioner.Provisioner,
 		if info.host.Status.ErrorCount > 0 {
 			desiredRebootMode = metal3api.RebootModeHard
 		}
-		provResult, err = prov.PowerOff(desiredRebootMode, info.host.Status.ErrorType == metal3api.PowerManagementError)
+		provResult, err = prov.PowerOff(desiredRebootMode, info.host.Status.ErrorType == metal3api.PowerManagementError, info.host.Spec.AutomatedCleaningMode)
 	}
 	if err != nil {
 		return actionError{fmt.Errorf("failed setting owner reference on hostUpdatePolicy: %w", err)}
diff --git a/pkg/provisioner/demo/demo.go b/pkg/provisioner/demo/demo.go
index 76ceb6af..f3cd5663 100644
--- a/pkg/provisioner/demo/demo.go
+++ b/pkg/provisioner/demo/demo.go
@@ -301,7 +301,7 @@ func (p *demoProvisioner) PowerOn(_ bool) (result provisioner.Result, err error)
 
 // PowerOff ensures the server is powered off independently of any image
 // provisioning operation.
-func (p *demoProvisioner) PowerOff(_ metal3api.RebootMode, _ bool) (result provisioner.Result, err error) {
+func (p *demoProvisioner) PowerOff(_ metal3api.RebootMode, _ bool, _ metal3api.AutomatedCleaningMode) (result provisioner.Result, err error) {
 	p.log.Info("powering off host")
 	return result, nil
 }
diff --git a/pkg/provisioner/fixture/fixture.go b/pkg/provisioner/fixture/fixture.go
index 046de785..4e7fd4e3 100644
--- a/pkg/provisioner/fixture/fixture.go
+++ b/pkg/provisioner/fixture/fixture.go
@@ -369,7 +369,7 @@ func (p *fixtureProvisioner) PowerOn(_ bool) (result provisioner.Result, err err
 
 // PowerOff ensures the server is powered off independently of any image
 // provisioning operation.
-func (p *fixtureProvisioner) PowerOff(_ metal3api.RebootMode, _ bool) (result provisioner.Result, err error) {
+func (p *fixtureProvisioner) PowerOff(_ metal3api.RebootMode, _ bool, _ metal3api.AutomatedCleaningMode) (result provisioner.Result, err error) {
 	p.log.Info("ensuring host is powered off")
 
 	if p.state.DisablePowerOff {
diff --git a/pkg/provisioner/ironic/ironic.go b/pkg/provisioner/ironic/ironic.go
index b52a78ba..0c05950d 100644
--- a/pkg/provisioner/ironic/ironic.go
+++ b/pkg/provisioner/ironic/ironic.go
@@ -1642,9 +1642,60 @@ func (p *ironicProvisioner) PowerOn(force bool) (result provisioner.Result, err
 	return result, nil
 }
 
+// abortInspectionOrCleaning aborts inspection or cleaning if the node is in one of those states.
+// This is necessary to allow power changes when the node is being deleted.
+// Abort only works in *Wait states; for *ing states we return Dirty to retry later.
+// For cleaning, we abort manual cleaning or automated cleaning if it's been disabled.
+func (p *ironicProvisioner) abortInspectionOrCleaning(ironicNode *nodes.Node, automatedCleaningMode metal3api.AutomatedCleaningMode) (result provisioner.Result, err error) {
+	provState := nodes.ProvisionState(ironicNode.ProvisionState)
+
+	switch provState {
+	case nodes.InspectWait:
+		p.log.Info("aborting inspection to allow power off during deletion")
+		return p.changeNodeProvisionState(
+			ironicNode,
+			nodes.ProvisionStateOpts{Target: nodes.TargetAbort},
+		)
+	case nodes.Inspecting:
+		p.log.Info("inspection in progress, waiting for it to reach inspect wait state")
+		return operationContinuing(provisionRequeueDelay)
+	case nodes.CleanWait:
+		// Abort manual cleaning or automated cleaning if disabled.
+		// Use automatedCleaningMode from BMH spec rather than ironicNode.AutomatedClean
+		// which may be stale if BMO couldn't update it during cleaning states.
+		isManualCleaning := ironicNode.TargetProvisionState == string(nodes.TargetClean)
+		if isManualCleaning || automatedCleaningMode == metal3api.CleaningModeDisabled {
+			if isManualCleaning {
+				p.log.Info("aborting manual cleaning to allow power off during deletion")
+			} else {
+				p.log.Info("aborting automated cleaning (disabled) to allow power off during deletion")
+			}
+			return p.changeNodeProvisionState(
+				ironicNode,
+				nodes.ProvisionStateOpts{Target: nodes.TargetAbort},
+			)
+		}
+		// Automated cleaning is enabled - let it finish
+		p.log.Info("automated cleaning in progress, waiting for it to complete")
+		return operationContinuing(provisionRequeueDelay)
+	case nodes.Cleaning:
+		// Check if it's manual cleaning or automated cleaning that's been disabled
+		if ironicNode.TargetProvisionState == string(nodes.TargetClean) ||
+			automatedCleaningMode == metal3api.CleaningModeDisabled {
+			p.log.Info("cleaning in progress, waiting for it to reach clean wait state before aborting")
+		} else {
+			p.log.Info("automated cleaning in progress, waiting for it to complete")
+		}
+		return operationContinuing(provisionRequeueDelay)
+	default:
+		// Node is not in a state that needs aborting
+		return operationComplete()
+	}
+}
+
 // PowerOff ensures the server is powered off independently of any image
 // provisioning operation.
-func (p *ironicProvisioner) PowerOff(rebootMode metal3api.RebootMode, force bool) (result provisioner.Result, err error) {
+func (p *ironicProvisioner) PowerOff(rebootMode metal3api.RebootMode, force bool, automatedCleaningMode metal3api.AutomatedCleaningMode) (result provisioner.Result, err error) {
 	p.log.Info(fmt.Sprintf("ensuring host is powered off (mode: %s)", rebootMode))
 
 	ironicNode, err := p.getNode()
@@ -1652,6 +1703,14 @@ func (p *ironicProvisioner) PowerOff(rebootMode metal3api.RebootMode, force bool
 		return transientError(err)
 	}
 
+	// If the node is in inspection or cleaning, we need to handle it before we can power off.
+	// For *Wait states, we can abort. For *ing states, we wait for them to transition to *Wait.
+	// This is especially important during deletion to avoid getting stuck waiting for inspection/cleaning to complete.
+	result, err = p.abortInspectionOrCleaning(ironicNode, automatedCleaningMode)
+	if err != nil || result.Dirty {
+		return result, err
+	}
+
 	if ironicNode.PowerState != powerOff {
 		targetState := ironicNode.TargetPowerState
 		// If the target state is either powerOff or softPowerOff, then we should wait
@@ -1684,7 +1743,7 @@ func (p *ironicProvisioner) PowerOff(rebootMode metal3api.RebootMode, force bool
 		return p.changePower(ironicNode, powerTarget)
 	}
 
-	return operationComplete()
+	return result, nil
 }
 
 func ironicNodeName(objMeta metav1.ObjectMeta) string {
diff --git a/pkg/provisioner/provisioner.go b/pkg/provisioner/provisioner.go
index 7358912c..787aaa46 100644
--- a/pkg/provisioner/provisioner.go
+++ b/pkg/provisioner/provisioner.go
@@ -196,7 +196,9 @@ type Provisioner interface {
 	// PowerOff ensures the server is powered off independently of any image
 	// provisioning operation. The boolean argument may be used to specify
 	// if a hard reboot (force power off) is required - true if so.
-	PowerOff(rebootMode metal3api.RebootMode, force bool) (result Result, err error)
+	// The automatedCleaningMode indicates the user's current intent regarding
+	// automated cleaning, used to determine if cleaning should be aborted during deletion.
+	PowerOff(rebootMode metal3api.RebootMode, force bool, automatedCleaningMode metal3api.AutomatedCleaningMode) (result Result, err error)
 
 	// TryInit checks if the provisioning backend is available to accept
 	// all the incoming requests and configures the available features.



Test patch:
diff --git a/internal/controller/metal3.io/host_state_machine_test.go b/internal/controller/metal3.io/host_state_machine_test.go
index e0c4ef22..0acf0428 100644
--- a/internal/controller/metal3.io/host_state_machine_test.go
+++ b/internal/controller/metal3.io/host_state_machine_test.go
@@ -1391,7 +1391,7 @@ func (m *mockProvisioner) PowerOn(_ bool) (result provisioner.Result, err error)
 	return m.getNextResultByMethod("PowerOn"), err
 }
 
-func (m *mockProvisioner) PowerOff(_ metal3api.RebootMode, _ bool) (result provisioner.Result, err error) {
+func (m *mockProvisioner) PowerOff(_ metal3api.RebootMode, _ bool, _ metal3api.AutomatedCleaningMode) (result provisioner.Result, err error) {
 	return m.getNextResultByMethod("PowerOff"), err
 }
 
diff --git a/pkg/provisioner/ironic/power_test.go b/pkg/provisioner/ironic/power_test.go
index 1f036a87..e7af6b5f 100644
--- a/pkg/provisioner/ironic/power_test.go
+++ b/pkg/provisioner/ironic/power_test.go
@@ -138,9 +138,10 @@ func TestPowerOff(t *testing.T) {
 	softPowerOffReason := "Host soft powered off"
 	nodeUUID := "33ce8659-7400-4c68-9535-d10766f07a58"
 	cases := []struct {
-		name   string
-		ironic *testserver.IronicMock
-		force  bool
+		name                  string
+		ironic                *testserver.IronicMock
+		force                 bool
+		automatedCleaningMode metal3api.AutomatedCleaningMode
 
 		expectedDirty        bool
 		expectedError        bool
@@ -245,6 +246,106 @@ func TestPowerOff(t *testing.T) {
 			expectedDirty:  true,
 			expectedReason: hardPowerOffReason,
 		},
+		{
+			name: "power-off while in InspectWait",
+			ironic: testserver.NewIronic(t).Node(nodes.Node{
+				PowerState:           powerOn,
+				ProvisionState:       string(nodes.InspectWait),
+				TargetProvisionState: string(nodes.TargetManage),
+				UUID:                 nodeUUID,
+			}).WithNodeStatesProvisionUpdate(nodeUUID),
+			rebootMode:           metal3api.RebootModeHard,
+			expectedDirty:        true,
+			expectedRequestAfter: 10,
+		},
+		{
+			name: "power-off while in Inspecting",
+			ironic: testserver.NewIronic(t).Node(nodes.Node{
+				PowerState:           powerOn,
+				ProvisionState:       string(nodes.Inspecting),
+				TargetProvisionState: string(nodes.TargetManage),
+				UUID:                 nodeUUID,
+			}).WithNodeStatesProvisionUpdate(nodeUUID),
+			rebootMode:           metal3api.RebootModeHard,
+			expectedDirty:        true,
+			expectedRequestAfter: 10,
+		},
+		{
+			name: "power-off while in CleanWait - manual cleaning",
+			ironic: testserver.NewIronic(t).Node(nodes.Node{
+				PowerState:           powerOn,
+				ProvisionState:       string(nodes.CleanWait),
+				TargetProvisionState: string(nodes.TargetClean),
+				UUID:                 nodeUUID,
+			}).WithNodeStatesProvisionUpdate(nodeUUID),
+			rebootMode:           metal3api.RebootModeHard,
+			expectedDirty:        true,
+			expectedRequestAfter: 10,
+		},
+		{
+			name: "power-off while in CleanWait - automated cleaning disabled",
+			ironic: testserver.NewIronic(t).Node(nodes.Node{
+				PowerState:           powerOn,
+				ProvisionState:       string(nodes.CleanWait),
+				TargetProvisionState: string(nodes.TargetManage),
+				UUID:                 nodeUUID,
+			}).WithNodeStatesProvisionUpdate(nodeUUID),
+			automatedCleaningMode: metal3api.CleaningModeDisabled,
+			rebootMode:            metal3api.RebootModeHard,
+			expectedDirty:         true,
+			expectedRequestAfter:  10,
+		},
+		{
+			name: "power-off while in CleanWait - automated cleaning enabled",
+			ironic: testserver.NewIronic(t).Node(nodes.Node{
+				PowerState:           powerOn,
+				ProvisionState:       string(nodes.CleanWait),
+				TargetProvisionState: string(nodes.TargetManage),
+				UUID:                 nodeUUID,
+			}),
+			automatedCleaningMode: metal3api.CleaningModeMetadata,
+			rebootMode:            metal3api.RebootModeHard,
+			expectedDirty:         true,
+			expectedRequestAfter:  10,
+		},
+		{
+			name: "power-off while in Cleaning - manual cleaning",
+			ironic: testserver.NewIronic(t).Node(nodes.Node{
+				PowerState:           powerOn,
+				ProvisionState:       string(nodes.Cleaning),
+				TargetProvisionState: string(nodes.TargetClean),
+				UUID:                 nodeUUID,
+			}),
+			rebootMode:           metal3api.RebootModeHard,
+			expectedDirty:        true,
+			expectedRequestAfter: 10,
+		},
+		{
+			name: "power-off while in Cleaning - automated cleaning disabled",
+			ironic: testserver.NewIronic(t).Node(nodes.Node{
+				PowerState:           powerOn,
+				ProvisionState:       string(nodes.Cleaning),
+				TargetProvisionState: string(nodes.TargetManage),
+				UUID:                 nodeUUID,
+			}),
+			automatedCleaningMode: metal3api.CleaningModeDisabled,
+			rebootMode:            metal3api.RebootModeHard,
+			expectedDirty:         true,
+			expectedRequestAfter:  10,
+		},
+		{
+			name: "power-off while in Cleaning - automated cleaning enabled",
+			ironic: testserver.NewIronic(t).Node(nodes.Node{
+				PowerState:           powerOn,
+				ProvisionState:       string(nodes.Cleaning),
+				TargetProvisionState: string(nodes.TargetManage),
+				UUID:                 nodeUUID,
+			}),
+			automatedCleaningMode: metal3api.CleaningModeMetadata,
+			rebootMode:            metal3api.RebootModeHard,
+			expectedDirty:         true,
+			expectedRequestAfter:  10,
+		},
 	}
 
 	for _, tc := range cases {
@@ -267,7 +368,7 @@ func TestPowerOff(t *testing.T) {
 			}
 
 			// We pass the RebootMode type here to define the reboot action
-			result, err := prov.PowerOff(tc.rebootMode, tc.force)
+			result, err := prov.PowerOff(tc.rebootMode, tc.force, tc.automatedCleaningMode)
 
 			assert.Equal(t, tc.expectedDirty, result.Dirty)
 			assert.Equal(t, time.Second*time.Duration(tc.expectedRequestAfter), result.RequeueAfter)
@@ -285,6 +386,14 @@ func TestPowerOff(t *testing.T) {
 			} else {
 				assert.Empty(t, eventReasons)
 			}
+
+			// Verify abort was called for InspectWait and CleanWait (manual/disabled) cases
+			if tc.name == "power-off while in InspectWait" ||
+				tc.name == "power-off while in CleanWait - manual cleaning" ||
+				tc.name == "power-off while in CleanWait - automated cleaning disabled" {
+				lastProvOp := tc.ironic.GetLastNodeStatesProvisionUpdateRequestFor(nodeUUID)
+				assert.Equal(t, nodes.TargetAbort, lastProvOp.Target, "expected abort to be called")
+			}
 		})
 	}
 }
@@ -311,7 +420,7 @@ func TestSoftPowerOffFallback(t *testing.T) {
 		t.Fatalf("could not create provisioner: %s", err)
 	}
 
-	_, err = prov.PowerOff(metal3api.RebootModeSoft, false)
+	_, err = prov.PowerOff(metal3api.RebootModeSoft, false, metal3api.CleaningModeMetadata)
 	require.Error(t, err)
 	assert.NotErrorAs(t, err, &softPowerOffUnsupportedError{})
 
diff --git a/test/e2e/common.go b/test/e2e/common.go
index 833383e0..6848e85d 100644
--- a/test/e2e/common.go
+++ b/test/e2e/common.go
@@ -142,6 +142,40 @@ func PatchBMHForProvisioning(ctx context.Context, input PatchBMHForProvisioningI
 	return helper.Patch(ctx, input.bmh)
 }
 
+// WaitForBmhReconciled waits for the BMO controller to process a BMH update.
+// This is used after BMO deployment rollout to ensure the controller is actually
+// processing events before making further changes. It works by adding/updating
+// an annotation and waiting for the status.lastUpdated timestamp to change,
+// which proves the controller reconciled the BMH.
+func WaitForBmhReconciled(ctx context.Context, c client.Client, bmh metal3api.BareMetalHost, intervals ...interface{}) {
+	// Get the current BMH state
+	key := types.NamespacedName{Namespace: bmh.Namespace, Name: bmh.Name}
+	currentBmh := &metal3api.BareMetalHost{}
+	Expect(c.Get(ctx, key, currentBmh)).To(Succeed())
+	initialLastUpdated := currentBmh.Status.LastUpdated
+
+	// Touch the BMH with an annotation update to trigger a reconcile
+	helper, err := patch.NewHelper(currentBmh, c)
+	Expect(err).NotTo(HaveOccurred())
+	if currentBmh.Annotations == nil {
+		currentBmh.Annotations = make(map[string]string)
+	}
+	currentBmh.Annotations["e2e.metal3.io/reconcile-check"] = metav1.Now().Format("2006-01-02T15:04:05Z")
+	Expect(helper.Patch(ctx, currentBmh)).To(Succeed())
+
+	// Wait for lastUpdated to change, proving the controller processed our update
+	Eventually(func(g Gomega) {
+		updatedBmh := &metal3api.BareMetalHost{}
+		g.Expect(c.Get(ctx, key, updatedBmh)).To(Succeed())
+		// Check that lastUpdated has changed (controller reconciled)
+		g.Expect(updatedBmh.Status.LastUpdated).NotTo(BeNil(), "BMH status.lastUpdated should not be nil")
+		if initialLastUpdated != nil {
+			g.Expect(updatedBmh.Status.LastUpdated.Before(initialLastUpdated)).To(BeFalse(),
+				"BMH status.lastUpdated should have been updated by controller")
+		}
+	}, intervals...).Should(Succeed())
+}
+
 // DeleteBmhsInNamespace deletes all BMHs in the given namespace.
 func DeleteBmhsInNamespace(ctx context.Context, deleter client.Client, namespace string) {
 	bmh := metal3api.BareMetalHost{}
diff --git a/test/e2e/e2e_config.go b/test/e2e/e2e_config.go
index 4687bd94..ba5e5f72 100644
--- a/test/e2e/e2e_config.go
+++ b/test/e2e/e2e_config.go
@@ -33,6 +33,18 @@ const (
 	ironicString string = "ironic"
 )
 
+// ContainerImage describes an image to load into a cluster.
+// This is a copy of clusterctl.ContainerImage with proper yaml tags for YAML unmarshaling.
+// See: https://github.com/metal3-io/baremetal-operator/issues/2832
+type ContainerImage struct {
+	// Name is the fully qualified name of the image.
+	Name string `yaml:"name"`
+
+	// LoadBehavior may be used to dictate whether a failed load operation
+	// should fail the test run.
+	LoadBehavior clusterctl.LoadImageBehavior `yaml:"loadBehavior,omitempty"`
+}
+
 type BMOIronicUpgradeInput struct {
 	// DeployIronic determines if Ironic should be installed at the beginning of the test.
 	// This should be generally set to `true`, but can be `false` in case Ironic is either pre-installed
@@ -57,13 +69,14 @@ type BMOIronicUpgradeInput struct {
 type Config struct {
 	// Images is a list of container images to load into the Kind cluster.
 	// Note that this not relevant when using an existing cluster.
-	Images []clusterctl.ContainerImage `json:"images,omitempty"`
+	// Uses custom ContainerImage type with yaml tags for proper YAML unmarshaling.
+	Images []ContainerImage `yaml:"images,omitempty"`
 
 	// Variables to be used in the tests.
-	Variables map[string]string `json:"variables,omitempty"`
+	Variables map[string]string `yaml:"variables,omitempty"`
 
 	// Intervals to be used for long operations during tests.
-	Intervals map[string][]string `json:"intervals,omitempty"`
+	Intervals map[string][]string `yaml:"intervals,omitempty"`
 
 	// BMOIronicUpgradeSpecs
 	BMOIronicUpgradeSpecs []BMOIronicUpgradeInput `yaml:"bmoIronicUpgradeSpecs,omitempty"`
@@ -100,6 +113,19 @@ func (c *Config) Defaults() {
 	}
 }
 
+// GetClusterctlImages converts the local ContainerImage slice to clusterctl.ContainerImage
+// for use with the CAPI bootstrap framework.
+func (c *Config) GetClusterctlImages() []clusterctl.ContainerImage {
+	result := make([]clusterctl.ContainerImage, len(c.Images))
+	for i, img := range c.Images {
+		result[i] = clusterctl.ContainerImage{
+			Name:         img.Name,
+			LoadBehavior: img.LoadBehavior,
+		}
+	}
+	return result
+}
+
 // Validate validates the configuration. More specifically:
 // - Image should have name and loadBehavior be one of [mustload, tryload].
 // - Intervals should be valid ginkgo intervals.
diff --git a/test/e2e/e2e_suite_test.go b/test/e2e/e2e_suite_test.go
index ca17f9a4..e376e647 100644
--- a/test/e2e/e2e_suite_test.go
+++ b/test/e2e/e2e_suite_test.go
@@ -88,7 +88,7 @@ var _ = SynchronizedBeforeSuite(func() []byte {
 	} else {
 		clusterProvider = bootstrap.CreateKindBootstrapClusterAndLoadImages(ctx, bootstrap.CreateKindBootstrapClusterAndLoadImagesInput{
 			Name:              "bmo-e2e",
-			Images:            e2eConfig.Images,
+			Images:            e2eConfig.GetClusterctlImages(),
 			ExtraPortMappings: e2eConfig.KindExtraPortMappings,
 		})
 		Expect(clusterProvider).ToNot(BeNil(), "Failed to create a cluster")
diff --git a/test/e2e/upgrade_test.go b/test/e2e/upgrade_test.go
index b04c86db..dc7a0370 100644
--- a/test/e2e/upgrade_test.go
+++ b/test/e2e/upgrade_test.go
@@ -319,6 +319,14 @@ func RunUpgradeTest(ctx context.Context, input *BMOIronicUpgradeInput, upgradeCl
 		})
 	}
 
+	// After deployment rollout, wait for the BMO controller to actually be processing events.
+	// There's a race condition where the deployment is "available" but the new pod hasn't
+	// acquired the leader lease yet or hasn't started its watches. If we patch the BMH
+	// during this window, the controller may miss the update event.
+	By("Waiting for BMO controller to be ready to process events")
+	WaitForBmhReconciled(ctx, upgradeClusterProxy.GetClient(), bmh,
+		e2eConfig.GetIntervals("default", "wait-deployment")...)
+
 	By("Patching the BMH to test provisioning")
 	// Using Eventually here since the webhook can take some time after the deployment is ready
 	Eventually(func() error {
@@ -365,7 +373,7 @@ var _ = Describe("Upgrade", Label("optional", "upgrade"), func() {
 			upgradeClusterName := fmt.Sprintf("bmo-e2e-upgrade-%d", GinkgoParallelProcess())
 			upgradeClusterProvider = bootstrap.CreateKindBootstrapClusterAndLoadImages(ctx, bootstrap.CreateKindBootstrapClusterAndLoadImagesInput{
 				Name:              upgradeClusterName,
-				Images:            e2eConfig.Images,
+				Images:            e2eConfig.GetClusterctlImages(),
 				ExtraPortMappings: e2eConfig.KindExtraPortMappings,
 			})
 			Expect(upgradeClusterProvider).ToNot(BeNil(), "Failed to create a cluster")
diff --git a/test/go.mod b/test/go.mod
index 7a8364a9..c09e1d58 100644
--- a/test/go.mod
+++ b/test/go.mod
@@ -16,9 +16,9 @@ require (
 	k8s.io/apimachinery v0.34.2
 	k8s.io/klog/v2 v2.130.1
 	k8s.io/utils v0.0.0-20250604170112-4c0f3b243397
-	libvirt.org/go/libvirt v1.11006.0
-	sigs.k8s.io/cluster-api v1.12.0-rc.0
-	sigs.k8s.io/cluster-api/test v1.12.0-rc.0
+	libvirt.org/go/libvirt v1.11010.0
+	sigs.k8s.io/cluster-api v1.12.0-rc.1
+	sigs.k8s.io/cluster-api/test v1.12.0-rc.1
 	sigs.k8s.io/controller-runtime v0.22.4
 	sigs.k8s.io/kind v0.30.0
 	sigs.k8s.io/kustomize/api v0.21.0
diff --git a/test/go.sum b/test/go.sum
index 7963b917..97d2c4f6 100644
--- a/test/go.sum
+++ b/test/go.sum
@@ -411,14 +411,14 @@ k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOP
 k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts=
 k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y=
 k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
-libvirt.org/go/libvirt v1.11006.0 h1:xzF87ptj/7cp1h4T62w1ZMBVY8m0mQukSCstMgeiVLs=
-libvirt.org/go/libvirt v1.11006.0/go.mod h1:1WiFE8EjZfq+FCVog+rvr1yatKbKZ9FaFMZgEqxEJqQ=
+libvirt.org/go/libvirt v1.11010.0 h1:1EIh2x6qcRoIBBOvrgN62vq5FIpgUBrmGadprQ/4M0Y=
+libvirt.org/go/libvirt v1.11010.0/go.mod h1:1WiFE8EjZfq+FCVog+rvr1yatKbKZ9FaFMZgEqxEJqQ=
 sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 h1:jpcvIRr3GLoUoEKRkHKSmGjxb6lWwrBlJsXc+eUYQHM=
 sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2/go.mod h1:Ve9uj1L+deCXFrPOk1LpFXqTg7LCFzFso6PA48q/XZw=
-sigs.k8s.io/cluster-api v1.12.0-rc.0 h1:7tWLgsGF/5zd0tdQloxdpBG1eEkYfgIFWFdYZaLVLF0=
-sigs.k8s.io/cluster-api v1.12.0-rc.0/go.mod h1:+S6WJdi8UPdqv5q9nka5al3ed/Qa0zAcSBgzTaa9VKA=
-sigs.k8s.io/cluster-api/test v1.12.0-rc.0 h1:xQuRM5nVY2dL8USqXRDnt4kT1MChK93BvpbQwonnNhc=
-sigs.k8s.io/cluster-api/test v1.12.0-rc.0/go.mod h1:eS05aBCEZeKzcW7MGnT9U+zap4NCpkxHBWb10DshWp0=
+sigs.k8s.io/cluster-api v1.12.0-rc.1 h1:ZoZtg/kRU4av5qt1z3MhbBP6hS1SPfWguFsbOiMX/Uc=
+sigs.k8s.io/cluster-api v1.12.0-rc.1/go.mod h1:+S6WJdi8UPdqv5q9nka5al3ed/Qa0zAcSBgzTaa9VKA=
+sigs.k8s.io/cluster-api/test v1.12.0-rc.1 h1:v/hXhN0mnIj8Z6gNk2Okt/XQycfRgvxDNNRcipS1xDA=
+sigs.k8s.io/cluster-api/test v1.12.0-rc.1/go.mod h1:eS05aBCEZeKzcW7MGnT9U+zap4NCpkxHBWb10DshWp0=
 sigs.k8s.io/controller-runtime v0.22.4 h1:GEjV7KV3TY8e+tJ2LCTxUTanW4z/FmNB7l327UfMq9A=
 sigs.k8s.io/controller-runtime v0.22.4/go.mod h1:+QX1XUpTXN4mLoblf4tqr5CQcyHPAki2HLXqQMY6vh8=
 sigs.k8s.io/gateway-api v1.1.0 h1:DsLDXCi6jR+Xz8/xd0Z1PYl2Pn0TyaFMOPPZIj4inDM=