Repository: metal3-io/baremetal-operator
Pull request #827 (2021-03-23 15:48:53)

Add support for detached annotation
This allows a new annotation as discussed in https://github.com/metal3-io/metal3-docs/pull/168

 ```
baremetalhost.metal3.io/unmanaged
```
*EDIT1* this has now been renamed to "deregistered" for both the annotation and OperationalStatus - I'll update the docs PR and this description when we've reached consensus on that new name.

*EDIT2* ...then renamed again to "detached"

When set this puts the BMH OperationalStatus into a "unmanaged",
which removes the host from the underlying provisioner (without
deprovisioning)

When the annotation removed the BMH can resume management without
any disruptive actions (the previous Provisioning State is maintained)

metal3-io-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/827#" title="Author self-approved">hardys</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)~~ [hardys]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
hardys: /cc @zaneb @dhellmann @andfasano 
hardys: /cc @maelk @fmuyassarov - this leaves the existing `paused` annotation behavior intact, and allows for a pivot where the BMH resources remain as inventory on the parent cluster.

In the community meeting we discussed the CAPI pivot process, can you confirm if this approach will be acceptable from that perspective since it's not impacting any existing APIs?
fmuyassarov: > /cc @maelk @fmuyassarov - this leaves the existing `paused` annotation behavior intact, and allows for a pivot where the BMH resources remain as inventory on the parent cluster.
> 
> In the community meeting we discussed the CAPI pivot process, can you confirm if this approach will be acceptable from that perspective since it's not impacting any existing APIs?

I believe it is okay, since this patch is not touching the pause annotation and logic in [`Reconcile()`](https://github.com/metal3-io/baremetal-operator/blob/0e6772bcef0ff4db9185b3ebaa6f3eeae58d6e60/controllers/metal3.io/baremetalhost_controller.go#L117).
/cc @kashifest 
kashifest: > /cc @maelk @fmuyassarov - this leaves the existing `paused` annotation behavior intact, and allows for a pivot where the BMH resources remain as inventory on the parent cluster.
> 
> In the community meeting we discussed the CAPI pivot process, can you confirm if this approach will be acceptable from that perspective since it's not impacting any existing APIs?

As long as `paused` and `status Annotation` behavior is intact, CAPI pivot process should be good IMO. 
hardys: @zaneb @andfasano I reworked this to the state machine as suggested, the new provisioner API (and docs/tests) are still TODO.

PTAL, if you're happy with this revised approach I'll proceed with the remaining pieces - thanks!
metal3-io-bot: @hardys: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/generate/1376576911869743105) | `/test generate`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @hardys: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/generate/1376576911869743105) | `/test generate`
govet | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/govet/1376576911869743111) | `/test govet`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @hardys: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/generate/1376576911869743105) | `/test generate`
govet | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/govet/1376576911869743111) | `/test govet`
unit | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/unit/1376576911869743106) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @hardys: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/generate/1376576911869743105) | `/test generate`
govet | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/govet/1376576911869743111) | `/test govet`
unit | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/unit/1376576911869743106) | `/test unit`
gofmt | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/gofmt/1376576911869743109) | `/test gofmt`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @hardys: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/generate/1376576911869743105) | `/test generate`
govet | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/govet/1376576911869743111) | `/test govet`
unit | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/unit/1376576911869743106) | `/test unit`
gofmt | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/gofmt/1376576911869743109) | `/test gofmt`
gomod | bc4f7b5862aeb1511ca16abbdfa70374450bce9e | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/gomod/1376576911869743112) | `/test gomod`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
hardys: /retitle Add support for detached annotation

hardys: /test-integration
hardys: @andfasano @zaneb this is ready for another review pass - PTAL thanks!

If we're OK with the latest name I'll update metal3-io/metal3-docs#168 to match?
hardys: Thanks for the review @zaneb I think I resolved all your comments - I also in discussion with @andfasano noticed that the "delete without deprovision" aspect wasn't working, so that's now resolved too and I added more tests to cover.
hardys: /test-integration
metal3-io-bot: @hardys: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/unit/1379362571051601926) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @hardys: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/unit/1379362571051601926) | `/test unit`
generate | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/generate/1379362571051601923) | `/test generate`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @hardys: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/unit/1379362571051601926) | `/test unit`
generate | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/generate/1379362571051601923) | `/test generate`
golint | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/golint/1379362571051601922) | `/test golint`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @hardys: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/unit/1379362571051601926) | `/test unit`
generate | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/generate/1379362571051601923) | `/test generate`
golint | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/golint/1379362571051601922) | `/test golint`
govet | bc54963cb8e08afcd857c5e6a9b894f07cbdf4ae | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/827/govet/1379362571051601925) | `/test govet`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
hardys: /test-integration
hardys: /test-integration
andfasano: /lgtm
/hold
to allow @zaneb a final pass 
metal3-io-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/827#" title="Author self-approved">hardys</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/827#pullrequestreview-630177054" title="Approved">zaneb</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)~~ [hardys,zaneb]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/apis/metal3.io/v1alpha1/baremetalhost_types.go b/apis/metal3.io/v1alpha1/baremetalhost_types.go
index 86ad9bd7..28a1de88 100644
--- a/apis/metal3.io/v1alpha1/baremetalhost_types.go
+++ b/apis/metal3.io/v1alpha1/baremetalhost_types.go
@@ -39,6 +39,10 @@ const (
 	// an immediate requeue)
 	PausedAnnotation = "baremetalhost.metal3.io/paused"
 
+	// Detached is the annotation which stops provisioner management of the host
+	// unlike in the paused case, the host status may be updated
+	DetachedAnnotation = "baremetalhost.metal3.io/detached"
+
 	// StatusAnnotation is the annotation that keeps a copy of the Status of BMH
 	// This is particularly useful when we pivot BMH. If the status
 	// annotation is present and status is empty, BMO will reconstruct BMH Status
@@ -121,6 +125,10 @@ const (
 	// OperationalStatusDelayed is the status value for when the host
 	// deployment needs to be delayed to limit simultaneous hosts provisioning
 	OperationalStatusDelayed = "delayed"
+
+	// OperationalStatusDetached is the status value when the host is
+	// marked unmanaged via the detached annotation
+	OperationalStatusDetached OperationalStatus = "detached"
 )
 
 // ErrorType indicates the class of problem that has caused the Host resource
@@ -147,6 +155,9 @@ const (
 	// PowerManagementError is an error condition occurring when the
 	// controller is unable to modify the power state of the Host.
 	PowerManagementError ErrorType = "power management error"
+	// DetachError is an error condition occurring when the
+	// controller is unable to detatch the host from the provisioner
+	DetachError ErrorType = "detach error"
 )
 
 // ProvisioningState defines the states the provisioner will report
@@ -625,7 +636,7 @@ type BareMetalHostStatus struct {
 	// after modifying this file
 
 	// OperationalStatus holds the status of the host
-	// +kubebuilder:validation:Enum="";OK;discovered;error;delayed
+	// +kubebuilder:validation:Enum="";OK;discovered;error;delayed;detached
 	OperationalStatus OperationalStatus `json:"operationalStatus"`
 
 	// ErrorType indicates the type of failure encountered when the
diff --git a/config/crd/bases/metal3.io_baremetalhosts.yaml b/config/crd/bases/metal3.io_baremetalhosts.yaml
index 43fa353d..3124c5b5 100644
--- a/config/crd/bases/metal3.io_baremetalhosts.yaml
+++ b/config/crd/bases/metal3.io_baremetalhosts.yaml
@@ -596,6 +596,7 @@ spec:
                 - discovered
                 - error
                 - delayed
+                - detached
                 type: string
               poweredOn:
                 description: indicator for whether or not the host is powered on
diff --git a/config/render/capm3.yaml b/config/render/capm3.yaml
index 5595c605..f28d16fc 100644
--- a/config/render/capm3.yaml
+++ b/config/render/capm3.yaml
@@ -594,6 +594,7 @@ spec:
                 - discovered
                 - error
                 - delayed
+                - detached
                 type: string
               poweredOn:
                 description: indicator for whether or not the host is powered on
diff --git a/controllers/metal3.io/baremetalhost_controller.go b/controllers/metal3.io/baremetalhost_controller.go
index 01f3ecfd..03ad0931 100644
--- a/controllers/metal3.io/baremetalhost_controller.go
+++ b/controllers/metal3.io/baremetalhost_controller.go
@@ -322,6 +322,7 @@ func recordActionFailure(info *reconcileInfo, errorType metal3v1alpha1.ErrorType
 	setErrorMessage(info.host, errorType, errorMessage)
 
 	eventType := map[metal3v1alpha1.ErrorType]string{
+		metal3v1alpha1.DetachError:                  "DetachError",
 		metal3v1alpha1.ProvisionedRegistrationError: "ProvisionedRegistrationError",
 		metal3v1alpha1.RegistrationError:            "RegistrationError",
 		metal3v1alpha1.InspectionError:              "InspectionError",
@@ -539,6 +540,33 @@ func getCurrentImage(host *metal3v1alpha1.BareMetalHost) *metal3v1alpha1.Image {
 	return nil
 }
 
+// detachHost() detaches the host from the Provisioner
+func (r *BareMetalHostReconciler) detachHost(prov provisioner.Provisioner, info *reconcileInfo) actionResult {
+	provResult, err := prov.Detach()
+	if err != nil {
+		return actionError{errors.Wrap(err, "failed to detach")}
+	}
+	if provResult.ErrorMessage != "" {
+		return recordActionFailure(info, metal3v1alpha1.DetachError, provResult.ErrorMessage)
+	}
+	if provResult.Dirty {
+		if info.host.Status.ErrorType == metal3v1alpha1.DetachError && clearError(info.host) {
+			return actionUpdate{actionContinue{provResult.RequeueAfter}}
+		}
+		return actionContinue{provResult.RequeueAfter}
+	}
+	slowPoll := actionContinue{unmanagedRetryDelay}
+	if info.host.Status.ErrorType == metal3v1alpha1.DetachError {
+		clearError(info.host)
+		info.host.Status.ErrorCount = 0
+	}
+	if info.host.SetOperationalStatus(metal3v1alpha1.OperationalStatusDetached) {
+		info.log.Info("host is detached, removed from provisioner")
+		return actionUpdate{slowPoll}
+	}
+	return slowPoll
+}
+
 // Test the credentials by connecting to the management controller.
 func (r *BareMetalHostReconciler) registerHost(prov provisioner.Provisioner, info *reconcileInfo) actionResult {
 	info.log.Info("registering and validating access to management controller",
diff --git a/controllers/metal3.io/host_state_machine.go b/controllers/metal3.io/host_state_machine.go
index 8476212d..8818756f 100644
--- a/controllers/metal3.io/host_state_machine.go
+++ b/controllers/metal3.io/host_state_machine.go
@@ -183,6 +183,10 @@ func (hsm *hostStateMachine) ReconcileState(info *reconcileInfo) (actionRes acti
 		return actionComplete{}
 	}
 
+	if detachedResult := hsm.checkDetachedHost(info); detachedResult != nil {
+		return detachedResult
+	}
+
 	if registerResult := hsm.ensureRegistered(info); registerResult != nil {
 		hostRegistrationRequired.Inc()
 		return registerResult
@@ -216,7 +220,11 @@ func (hsm *hostStateMachine) checkInitiateDelete() bool {
 	default:
 		hsm.NextState = metal3v1alpha1.StateDeleting
 	case metal3v1alpha1.StateProvisioning, metal3v1alpha1.StateProvisioned:
-		hsm.NextState = metal3v1alpha1.StateDeprovisioning
+		if hsm.Host.OperationalStatus() == metal3v1alpha1.OperationalStatusDetached {
+			hsm.NextState = metal3v1alpha1.StateDeleting
+		} else {
+			hsm.NextState = metal3v1alpha1.StateDeprovisioning
+		}
 	case metal3v1alpha1.StateDeprovisioning:
 		// Allow state machine to run to continue deprovisioning.
 		return false
@@ -227,6 +235,46 @@ func (hsm *hostStateMachine) checkInitiateDelete() bool {
 	return true
 }
 
+// hasInspectAnnotation checks for existence of baremetalhost.metal3.io/detached
+func hasDetachedAnnotation(host *metal3v1alpha1.BareMetalHost) bool {
+	annotations := host.GetAnnotations()
+	if annotations != nil {
+		if _, ok := annotations[metal3v1alpha1.DetachedAnnotation]; ok {
+			return true
+		}
+	}
+	return false
+}
+
+func (hsm *hostStateMachine) checkDetachedHost(info *reconcileInfo) (result actionResult) {
+	// If the detached annotation is set we remove any host from the
+	// provisioner and take no further action
+	// Note this doesn't change the current state, only the OperationalStatus
+	if hasDetachedAnnotation(hsm.Host) {
+		// Only allow detaching hosts in Provisioned/ExternallyProvisioned states
+		switch info.host.Status.Provisioning.State {
+		case metal3v1alpha1.StateProvisioned, metal3v1alpha1.StateExternallyProvisioned:
+			return hsm.Reconciler.detachHost(hsm.Provisioner, info)
+		}
+	}
+	if info.host.Status.ErrorType == metal3v1alpha1.DetachError {
+		clearError(info.host)
+		hsm.Host.Status.ErrorCount = 0
+		info.log.Info("removed detach error")
+		return actionUpdate{}
+	}
+	if info.host.OperationalStatus() == metal3v1alpha1.OperationalStatusDetached {
+		newStatus := metal3v1alpha1.OperationalStatusOK
+		if info.host.Status.ErrorType != "" {
+			newStatus = metal3v1alpha1.OperationalStatusError
+		}
+		info.host.SetOperationalStatus(newStatus)
+		info.log.Info("removed detached status")
+		return actionUpdate{}
+	}
+	return nil
+}
+
 func (hsm *hostStateMachine) ensureRegistered(info *reconcileInfo) (result actionResult) {
 	if !hsm.haveCreds {
 		// If we are in the process of deletion (which may start with
diff --git a/docs/api.md b/docs/api.md
index 99d9065c..b12cf2ae 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -459,3 +459,14 @@ put any value on this annotation **other than `metal3.io/capm3`**. Please make
 sure that you remove the annotation  **only if the value of the annotation is
 not `metal3.io/capm3`, but another value that you have provided**. Removing the
 annotation will enable the reconciliation again.
+
+## Detaching hosts
+
+It is possible to prevent management of a BareMetalHost object by adding
+an annotation `baremetalhost.metal3.io/detached`. This removes the host from
+the provisioner, which prevents any management of the physical host (e.g
+changing power state, or deprovisioning), but still allows the BMH status
+to be updated unlike the `paused` anotation. While in this state the
+OperationalStatus field will be `detached` but the provisioning state will
+be unmodified.  This API only has any effect for BareMetalHost resources
+that are in either `Provisioned` or `ExternallyProvisioned` state.
diff --git a/pkg/provisioner/demo/demo.go b/pkg/provisioner/demo/demo.go
index 96b1f8ab..6714f47d 100644
--- a/pkg/provisioner/demo/demo.go
+++ b/pkg/provisioner/demo/demo.go
@@ -288,6 +288,16 @@ func (p *demoProvisioner) Delete() (result provisioner.Result, err error) {
 	return result, nil
 }
 
+// Detach removes the host from the provisioning system.
+// Similar to Delete, but ensures non-interruptive behavior
+// for the target system.  It may be called multiple times,
+// and should return true for its dirty  flag until the
+// deletion operation is completed.
+func (p *demoProvisioner) Detach() (result provisioner.Result, err error) {
+	p.log.Info("detaching host")
+	return result, nil
+}
+
 // PowerOn ensures the server is powered on independently of any image
 // provisioning operation.
 func (p *demoProvisioner) PowerOn() (result provisioner.Result, err error) {
diff --git a/pkg/provisioner/fixture/fixture.go b/pkg/provisioner/fixture/fixture.go
index ba3ff816..1bbaf378 100644
--- a/pkg/provisioner/fixture/fixture.go
+++ b/pkg/provisioner/fixture/fixture.go
@@ -259,6 +259,15 @@ func (p *fixtureProvisioner) Delete() (result provisioner.Result, err error) {
 	return result, nil
 }
 
+// Detach removes the host from the provisioning system.
+// Similar to Delete, but ensures non-interruptive behavior
+// for the target system.  It may be called multiple times,
+// and should return true for its dirty  flag until the
+// deletion operation is completed.
+func (p *fixtureProvisioner) Detach() (result provisioner.Result, err error) {
+	return p.Delete()
+}
+
 // PowerOn ensures the server is powered on independently of any image
 // provisioning operation.
 func (p *fixtureProvisioner) PowerOn() (result provisioner.Result, err error) {
diff --git a/pkg/provisioner/ironic/ironic.go b/pkg/provisioner/ironic/ironic.go
index 183c5ed1..c1c5086a 100644
--- a/pkg/provisioner/ironic/ironic.go
+++ b/pkg/provisioner/ironic/ironic.go
@@ -1665,6 +1665,16 @@ func (p *ironicProvisioner) Delete() (result provisioner.Result, err error) {
 	return operationContinuing(0)
 }
 
+// Detach removes the host from the provisioning system.
+// Similar to Delete, but ensures non-interruptive behavior
+// for the target system.  It may be called multiple times,
+// and should return true for its dirty  flag until the
+// deletion operation is completed.
+func (p *ironicProvisioner) Detach() (result provisioner.Result, err error) {
+	// Currently the same behavior as Delete()
+	return p.Delete()
+}
+
 func (p *ironicProvisioner) changePower(ironicNode *nodes.Node, target nodes.TargetPowerState) (result provisioner.Result, err error) {
 	p.log.Info("changing power state")
 
diff --git a/pkg/provisioner/provisioner.go b/pkg/provisioner/provisioner.go
index 5422208d..0e70e821 100644
--- a/pkg/provisioner/provisioner.go
+++ b/pkg/provisioner/provisioner.go
@@ -130,6 +130,13 @@ type Provisioner interface {
 	// flag until the deletion operation is completed.
 	Delete() (result Result, err error)
 
+	// Detach removes the host from the provisioning system.
+	// Similar to Delete, but ensures non-interruptive behavior
+	// for the target system.  It may be called multiple times,
+	// and should return true for its dirty  flag until the
+	// deletion operation is completed.
+	Detach() (result Result, err error)
+
 	// PowerOn ensures the server is powered on independently of any image
 	// provisioning operation.
 	PowerOn() (result Result, err error)


Test patch:
diff --git a/controllers/metal3.io/host_state_machine_test.go b/controllers/metal3.io/host_state_machine_test.go
index e6ff6f1c..595b23b0 100644
--- a/controllers/metal3.io/host_state_machine_test.go
+++ b/controllers/metal3.io/host_state_machine_test.go
@@ -146,6 +146,238 @@ func TestProvisioningCapacity(t *testing.T) {
 	}
 }
 
+func TestDetach(t *testing.T) {
+	testCases := []struct {
+		Scenario                  string
+		Host                      *metal3v1alpha1.BareMetalHost
+		HasDetachedAnnotation     bool
+		ExpectedDetach            bool
+		ExpectedDirty             bool
+		ExpectedOperationalStatus metal3v1alpha1.OperationalStatus
+		ExpectedState             metal3v1alpha1.ProvisioningState
+	}{
+		{
+			Scenario:                  "ProvisionedHost",
+			Host:                      host(metal3v1alpha1.StateProvisioned).build(),
+			ExpectedDetach:            false,
+			ExpectedDirty:             false,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateProvisioned,
+		},
+		{
+			Scenario:                  "DetachProvisionedHost",
+			Host:                      host(metal3v1alpha1.StateProvisioned).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            true,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusDetached,
+			ExpectedState:             metal3v1alpha1.StateProvisioned,
+		},
+		{
+			Scenario:                  "DeleteDetachedProvisionedHost",
+			Host:                      host(metal3v1alpha1.StateProvisioned).SetOperationalStatus(metal3v1alpha1.OperationalStatusDetached).setDeletion().build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusDetached,
+			// Should move to Deleting without any Deprovisioning
+			ExpectedState: metal3v1alpha1.StateDeleting,
+		},
+		{
+			Scenario:                  "ExternallyProvisionedHost",
+			Host:                      host(metal3v1alpha1.StateExternallyProvisioned).SetExternallyProvisioned().build(),
+			HasDetachedAnnotation:     false,
+			ExpectedDetach:            false,
+			ExpectedDirty:             false,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateExternallyProvisioned,
+		},
+		{
+			Scenario:                  "DetachExternallyProvisionedHost",
+			Host:                      host(metal3v1alpha1.StateExternallyProvisioned).SetExternallyProvisioned().build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            true,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusDetached,
+			ExpectedState:             metal3v1alpha1.StateExternallyProvisioned,
+		},
+		{
+			Scenario:                  "NoneHost",
+			Host:                      host(metal3v1alpha1.StateNone).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusDiscovered,
+			ExpectedState:             metal3v1alpha1.StateUnmanaged,
+		},
+		{
+			Scenario:                  "UnmanagedHost",
+			Host:                      host(metal3v1alpha1.StateUnmanaged).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             false,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateUnmanaged,
+		},
+		{
+			Scenario:                  "RegisteringHost",
+			Host:                      host(metal3v1alpha1.StateRegistering).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateInspecting,
+		},
+		{
+			Scenario:                  "InspectingHost",
+			Host:                      host(metal3v1alpha1.StateInspecting).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateMatchProfile,
+		},
+		{
+			Scenario:                  "MatchProfileHost",
+			Host:                      host(metal3v1alpha1.StateMatchProfile).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StatePreparing,
+		},
+		{
+			Scenario:                  "AvailableHost",
+			Host:                      host(metal3v1alpha1.StateAvailable).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StatePreparing,
+		},
+		{
+			Scenario:                  "PreparingHost",
+			Host:                      host(metal3v1alpha1.StatePreparing).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateReady,
+		},
+		{
+			Scenario:                  "ReadyHost",
+			Host:                      host(metal3v1alpha1.StateReady).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StatePreparing,
+		},
+		{
+			Scenario:                  "ProvisioningHost",
+			Host:                      host(metal3v1alpha1.StateProvisioning).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateProvisioned,
+		},
+		{
+			Scenario:                  "DeprovisioningHost",
+			Host:                      host(metal3v1alpha1.StateDeprovisioning).build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             true,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateReady,
+		},
+		{
+			Scenario:                  "DeletingHost",
+			Host:                      host(metal3v1alpha1.StateDeleting).setDeletion().build(),
+			HasDetachedAnnotation:     true,
+			ExpectedDetach:            false,
+			ExpectedDirty:             false,
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateDeleting,
+		},
+	}
+	for _, tc := range testCases {
+		t.Run(tc.Scenario, func(t *testing.T) {
+			if tc.HasDetachedAnnotation {
+				tc.Host.Annotations = map[string]string{
+					metal3v1alpha1.DetachedAnnotation: "true",
+				}
+			}
+			prov := newMockProvisioner()
+			hsm := newHostStateMachine(tc.Host, &BareMetalHostReconciler{}, prov, true)
+			info := makeDefaultReconcileInfo(tc.Host)
+			result := hsm.ReconcileState(info)
+
+			assert.Equal(t, tc.ExpectedDetach, prov.calledNoError("Detach"), "ExpectedDetach mismatch")
+			assert.Equal(t, tc.ExpectedDirty, result.Dirty(), "ExpectedDirty mismatch")
+			assert.Equal(t, tc.ExpectedOperationalStatus, info.host.OperationalStatus())
+			assert.Equal(t, tc.ExpectedState, info.host.Status.Provisioning.State)
+		})
+	}
+}
+
+func TestDetachError(t *testing.T) {
+	testCases := []struct {
+		Scenario                  string
+		Host                      *metal3v1alpha1.BareMetalHost
+		ExpectedOperationalStatus metal3v1alpha1.OperationalStatus
+		ExpectedState             metal3v1alpha1.ProvisioningState
+		ClearError                bool
+		RemoveAnnotation          bool
+	}{
+		{
+			Scenario:                  "ProvisionerTemporaryError",
+			Host:                      host(metal3v1alpha1.StateProvisioned).build(),
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusDetached,
+			ExpectedState:             metal3v1alpha1.StateProvisioned,
+			ClearError:                true,
+		},
+		{
+			Scenario:                  "AnnotationRemovedAfterDetachError",
+			Host:                      host(metal3v1alpha1.StateProvisioned).build(),
+			ExpectedOperationalStatus: metal3v1alpha1.OperationalStatusOK,
+			ExpectedState:             metal3v1alpha1.StateProvisioned,
+			RemoveAnnotation:          true,
+		},
+	}
+	for _, tc := range testCases {
+		t.Run(tc.Scenario, func(t *testing.T) {
+			tc.Host.Annotations = map[string]string{
+				metal3v1alpha1.DetachedAnnotation: "true",
+			}
+			prov := newMockProvisioner()
+			hsm := newHostStateMachine(tc.Host, &BareMetalHostReconciler{}, prov, true)
+			info := makeDefaultReconcileInfo(tc.Host)
+
+			prov.setNextError("Detach", "some error")
+			result := hsm.ReconcileState(info)
+			assert.True(t, result.Dirty())
+			assert.Equal(t, 1, tc.Host.Status.ErrorCount)
+			assert.Equal(t, metal3v1alpha1.OperationalStatusError, info.host.OperationalStatus())
+			assert.Equal(t, v1alpha1.DetachError, info.host.Status.ErrorType)
+			assert.Equal(t, tc.ExpectedState, info.host.Status.Provisioning.State)
+
+			if tc.ClearError {
+				prov.clearNextError("Detach")
+			}
+			if tc.RemoveAnnotation {
+				tc.Host.Annotations = map[string]string{}
+			}
+			result = hsm.ReconcileState(info)
+			assert.Equal(t, 0, tc.Host.Status.ErrorCount)
+			assert.True(t, result.Dirty())
+			assert.Equal(t, tc.ExpectedOperationalStatus, info.host.OperationalStatus())
+			assert.Equal(t, tc.ExpectedState, info.host.Status.Provisioning.State)
+			assert.Empty(t, info.host.Status.ErrorType)
+		})
+	}
+}
+
 func TestProvisioningCancelled(t *testing.T) {
 	testCases := []struct {
 		Scenario string
@@ -587,17 +819,21 @@ func newMockProvisioner() *mockProvisioner {
 	return &mockProvisioner{
 		hasProvisioningCapacity: true,
 		nextResults:             make(map[string]provisioner.Result),
+		callsNoError:            make(map[string]bool),
 	}
 }
 
 type mockProvisioner struct {
 	hasProvisioningCapacity bool
 	nextResults             map[string]provisioner.Result
+	callsNoError            map[string]bool
 }
 
 func (m *mockProvisioner) getNextResultByMethod(name string) (result provisioner.Result) {
 	if value, ok := m.nextResults[name]; ok {
 		result = value
+	} else {
+		m.callsNoError[name] = true
 	}
 	return
 }
@@ -616,6 +852,14 @@ func (m *mockProvisioner) setNextError(methodName, msg string) {
 	}
 }
 
+func (m *mockProvisioner) clearNextError(methodName string) {
+	m.nextResults[methodName] = provisioner.Result{}
+}
+
+func (m *mockProvisioner) calledNoError(methodName string) bool {
+	return m.callsNoError[methodName]
+}
+
 func (m *mockProvisioner) ValidateManagementAccess(data provisioner.ManagementAccessData, credentialsChanged, force bool) (result provisioner.Result, provID string, err error) {
 	return m.getNextResultByMethod("ValidateManagementAccess"), "", err
 }
@@ -649,6 +893,11 @@ func (m *mockProvisioner) Delete() (result provisioner.Result, err error) {
 	return m.getNextResultByMethod("Delete"), err
 }
 
+func (m *mockProvisioner) Detach() (result provisioner.Result, err error) {
+	res := m.getNextResultByMethod("Detach")
+	return res, err
+}
+
 func (m *mockProvisioner) PowerOn() (result provisioner.Result, err error) {
 	return m.getNextResultByMethod("PowerOn"), err
 }
diff --git a/pkg/provisioner/ironic/delete_test.go b/pkg/provisioner/ironic/delete_test.go
index c456dd34..0b212efa 100644
--- a/pkg/provisioner/ironic/delete_test.go
+++ b/pkg/provisioner/ironic/delete_test.go
@@ -9,11 +9,22 @@ import (
 	"github.com/stretchr/testify/assert"
 
 	"github.com/metal3-io/baremetal-operator/pkg/bmc"
+	"github.com/metal3-io/baremetal-operator/pkg/provisioner"
 	"github.com/metal3-io/baremetal-operator/pkg/provisioner/ironic/clients"
 	"github.com/metal3-io/baremetal-operator/pkg/provisioner/ironic/testserver"
 )
 
+type TestFunc func(string)
+
 func TestDelete(t *testing.T) {
+	deleteTest(t, false)
+}
+
+func TestDetach(t *testing.T) {
+	deleteTest(t, true)
+}
+
+func deleteTest(t *testing.T, detach bool) {
 
 	nodeUUID := "33ce8659-7400-4c68-9535-d10766f07a58"
 
@@ -172,7 +183,12 @@ func TestDelete(t *testing.T) {
 				t.Fatalf("could not create provisioner: %s", err)
 			}
 
-			result, err := prov.Delete()
+			var result provisioner.Result
+			if detach {
+				result, err = prov.Detach()
+			} else {
+				result, err = prov.Delete()
+			}
 
 			assert.Equal(t, tc.expectedDirty, result.Dirty)
 			assert.Equal(t, tc.expectedRequestAfter, result.RequeueAfter)