Repository: metal3-io/baremetal-operator
Pull request #807 (2021-03-03 20:55:38)

Use PROVISIONING_LIMIT also for deprovisioning.
Signed-off-by: s3rj1k <evasive.gyron@gmail.com>

metal3-io-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/807#" title="Author self-approved">s3rj1k</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **dhellmann**
You can assign the PR to them by writing `/assign @dhellmann` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["dhellmann"]} -->
metal3-io-bot: Hi @s3rj1k. Thanks for your PR.

I'm waiting for a [metal3-io](https://github.com/orgs/metal3-io/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://github.com/orgs/metal3-io/people) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

s3rj1k: /assign @dhellmann
dhellmann: I like the general idea of doing this, but I think Andrea is a better person to approve the implementation, since he's more familiar with how that part of the system works.

/ok-to-test
/unassign @dhellmann
/assign @andfasano 
s3rj1k: This adds limit for deleting/deprovisioning host to PROVISIONING_LIMIT.

Verified manually, works, needs work for unit tests.

@dhellmann Please look at this, if this is feasible solution I will fix unit tests in following commits, thanks.
s3rj1k: @dhellmann Great! Thanks! That was super quick answer)
metal3-io-bot: @s3rj1k: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | b059c4dec2c75537160c839a67d398bd30a2317a | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/unit/1367217793094651905) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @s3rj1k: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | b059c4dec2c75537160c839a67d398bd30a2317a | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/unit/1367217793098846209) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
s3rj1k: /retest
metal3-io-bot: @s3rj1k: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | ce07ced1fb6bd79e538c487d1a922c412f29b042 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/unit/1367232391025790977) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
s3rj1k: /retest
s3rj1k: @andfasano Can you please look at this? Specifically unit-tests
andfasano: > @andfasano Can you please look at this? Specifically unit-tests

@s3rj1k I'd like first to understand the motivation if possible, because this feature was originally linked to this proposal https://github.com/metal3-io/metal3-docs/blob/master/design/baremetal-operator/limit-hosts-provisioning.md, where we limited the scope just for the provisioning phase - whereas there was no interest in limiting the activity during the deprovisioning. Did you maybe have any specific scenario to be addressed?
s3rj1k: > > @andfasano Can you please look at this? Specifically unit-tests
> 
> @s3rj1k I'd like first to understand the motivation if possible, because this feature was originally linked to this proposal https://github.com/metal3-io/metal3-docs/blob/master/design/baremetal-operator/limit-hosts-provisioning.md, where we limited the scope just for the provisioning phase - whereas there was no interest in limiting the activity during the deprovisioning. Did you maybe have any specific scenario to be addressed?

When deleting a big cluster (lot's of bmhs) I see similar issues (similar to provisioning lots of hosts simultaneously) with DHCP server during deprovisioning. 
s3rj1k: Also docs have `deleting` in pseudo-code example (https://github.com/metal3-io/metal3-docs/blob/master/design/baremetal-operator/limit-hosts-provisioning.md)
![изображение](https://user-images.githubusercontent.com/11349489/110006704-a4ab9380-7d22-11eb-82d4-fd9e30e27a93.png)

s3rj1k: @andfasano ?
metal3-io-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/807#pullrequestreview-606436059" title="Approved">dtantsur</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/807#" title="Author self-approved">s3rj1k</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **dhellmann**
You can assign the PR to them by writing `/assign @dhellmann` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["dhellmann"]} -->
s3rj1k: @dtantsur fixed, thanks
Can we trigger `test-integration` on this PR?
dhellmann: /test-integration
s3rj1k: @andfasano How about renaming Prometheus counter metal3_delayed__provisioning_total -> metal3_delayed__total similar to first points?

> As a last comment, probably such enhancement could be worth to have a related design proposal - or amend accordingly the existing one?

Better to amend existing one.
dhellmann: > > @andfasano How about renaming Prometheus counter metal3_delayed__provisioning_total -> metal3_delayed__total similar to first points?
> 
> IMO it would make more sense to have two separate counters to track two different phases (provisioning and deprovisioning)

We should also consider whether we count the metrics as part of our API. If we do, then renaming something becomes non-trivial, and adding a new one is simpler.
s3rj1k: In addition renaming metric makes PR trivial, adding newer metric would make PR a bit bigger
andfasano: > Also as a note, @dtantsur pointed out that `nodes.Deleting` is a short phase and actual work is done in CleanWait [#807 (comment)](https://github.com/metal3-io/baremetal-operator/pull/807#discussion_r589530082)
> 
> So the question is, how reliable new deprovision metric would be?
> 
> cc: @dhellmann @andfasano

I think that original discussion focused on including all the `*Wait` and `*ing` states from the [Ironic states](https://docs.openstack.org/ironic/latest/_images/states.svg), so a possibility could be to reintroduce the missing ones. 
s3rj1k: So the main question that needs to be answered to know how we are going to handle this is:

> We should also consider whether we count the metrics as part of our API.

cc: @dhellmann @andfasano @dtantsur 
andfasano: > So the main question that needs to be answered to know how we are going to handle this is:
> 
> > We should also consider whether we count the metrics as part of our API.
> 
> cc: @dhellmann @andfasano @dtantsur

The metric is something that popped out during the development phase, but I think it could make sense to consider it part of the initial API. 
s3rj1k: @andfasano How about we rename metric to maintain simplicity of change in this PR and then update design docs to state that this (renamed) metric is part of API?

That way we could merge this change without refactoring nodes states that @dtantsur mentioned earlier.

cc: @dhellmann 
dhellmann: > @andfasano How about we rename metric to maintain simplicity of change in this PR and then update design docs to state that this (renamed) metric is part of API?
> 
> That way we could merge this change without refactoring nodes states that @dtantsur mentioned earlier.
> 
> cc: @dhellmann

I have customers in production with this version, and while I don't know if any of them look at the metric I do consider it to be part of the API. So, we need to maintain the existing metric as-is, and either add a new more generic one that counts for both cases or add a new one just for deprovisioning.
metal3-io-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/807#pullrequestreview-606436059" title="Approved">dtantsur</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/807#" title="Author self-approved">s3rj1k</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)~~ [dtantsur]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
metal3-io-bot: @s3rj1k: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/unit/1374027596144054275) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @s3rj1k: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/unit/1374027596144054275) | `/test unit`
generate | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/generate/1374027596144054278) | `/test generate`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @s3rj1k: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/unit/1374027596144054275) | `/test unit`
generate | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/generate/1374027596144054278) | `/test generate`
govet | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/govet/1374027596144054274) | `/test govet`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @s3rj1k: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/unit/1374027596144054275) | `/test unit`
generate | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/generate/1374027596144054278) | `/test generate`
govet | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/govet/1374027596144054274) | `/test govet`
gosec | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/gosec/1374027596144054273) | `/test gosec`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
s3rj1k: /test unit
metal3-io-bot: @s3rj1k: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/generate/1374027596144054278) | `/test generate`
govet | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/govet/1374027596144054274) | `/test govet`
gosec | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/gosec/1374027596144054273) | `/test gosec`
unit | 32b22186405d900e91083e3b2b57cc0a455caa61 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/807/unit/1374030817344360448) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
s3rj1k: /test unit
s3rj1k: /test-integration
s3rj1k: @andfasano Addressed your previous points and refactored this PR, please take a look. 
s3rj1k: /test-integration
s3rj1k: @andfasano Rebased PR, please take a look.
s3rj1k: /test-integration
s3rj1k: @andfasano Refactored based on your previous points, please take a look again, thanks.
metal3-io-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/807#pullrequestreview-625146744" title="Approved">andfasano</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/807#pullrequestreview-606436059" title="Approved">dtantsur</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/807#" title="Author self-approved">s3rj1k</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)~~ [dtantsur]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
andfasano: /test-integration
andfasano: @s3rj1k looks like some conflicts needs to be resolved to proceed
s3rj1k: /test-integration
s3rj1k: @andfasano Fixed, all tests are green
andfasano: @s3rj1k Can you please squash the commits?

Patch:
diff --git a/controllers/metal3.io/baremetalhost_controller.go b/controllers/metal3.io/baremetalhost_controller.go
index 03ad0931..6f651016 100644
--- a/controllers/metal3.io/baremetalhost_controller.go
+++ b/controllers/metal3.io/baremetalhost_controller.go
@@ -27,6 +27,7 @@ import (
 
 	"github.com/go-logr/logr"
 	"github.com/pkg/errors"
+	"github.com/prometheus/client_golang/prometheus"
 
 	corev1 "k8s.io/api/core/v1"
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
@@ -338,9 +339,16 @@ func recordActionFailure(info *reconcileInfo, errorType metal3v1alpha1.ErrorType
 	return actionFailed{dirty: true, ErrorType: errorType, errorCount: info.host.Status.ErrorCount}
 }
 
-func recordActionDelayed(info *reconcileInfo) actionResult {
+func recordActionDelayed(info *reconcileInfo, state metal3v1alpha1.ProvisioningState) actionResult {
+	var counter prometheus.Counter
+
+	switch state {
+	case metal3v1alpha1.StateDeprovisioning, metal3v1alpha1.StateDeleting:
+		counter = delayedDeprovisioningHostCounters.With(hostMetricLabels(info.request))
+	default:
+		counter = delayedProvisioningHostCounters.With(hostMetricLabels(info.request))
+	}
 
-	counter := delayedProvisioningHostCounters.With(hostMetricLabels(info.request))
 	info.postSaveCallbacks = append(info.postSaveCallbacks, counter.Inc)
 
 	info.host.SetOperationalStatus(metal3v1alpha1.OperationalStatusDelayed)
diff --git a/controllers/metal3.io/host_state_machine.go b/controllers/metal3.io/host_state_machine.go
index 8818756f..a35ce906 100644
--- a/controllers/metal3.io/host_state_machine.go
+++ b/controllers/metal3.io/host_state_machine.go
@@ -79,13 +79,14 @@ func recordStateEnd(info *reconcileInfo, host *metal3v1alpha1.BareMetalHost, sta
 	return
 }
 
-func (hsm *hostStateMachine) ensureProvisioningCapacity(info *reconcileInfo) actionResult {
-	hasCapacity, err := hsm.Provisioner.HasProvisioningCapacity()
+func (hsm *hostStateMachine) ensureCapacity(info *reconcileInfo, state metal3v1alpha1.ProvisioningState) actionResult {
+	hasCapacity, err := hsm.Provisioner.HasCapacity()
 	if err != nil {
-		return actionError{errors.Wrap(err, "failed to get hosts currently being provisioned")}
+		return actionError{errors.Wrap(err, "failed to determine current provisioner capacity")}
 	}
+
 	if !hasCapacity {
-		return recordActionDelayed(info)
+		return recordActionDelayed(info, state)
 	}
 
 	return nil
@@ -96,12 +97,13 @@ func (hsm *hostStateMachine) updateHostStateFrom(initialState metal3v1alpha1.Pro
 	if hsm.NextState != initialState {
 
 		// Check if there is a free slot available when trying to
-		// provision an host - if not the action will be delayed.
-		// The check is limited to only the provisioning states to
+		// (de)provision an host - if not the action will be delayed.
+		// The check is limited to only the (de)provisioning states to
 		// avoid putting an excessive pressure on the provisioner
 		switch hsm.NextState {
-		case metal3v1alpha1.StateInspecting, metal3v1alpha1.StateProvisioning:
-			if actionRes := hsm.ensureProvisioningCapacity(info); actionRes != nil {
+		case metal3v1alpha1.StateInspecting, metal3v1alpha1.StateProvisioning,
+			metal3v1alpha1.StateDeprovisioning, metal3v1alpha1.StateDeleting:
+			if actionRes := hsm.ensureCapacity(info, hsm.NextState); actionRes != nil {
 				return actionRes
 			}
 		}
@@ -144,7 +146,7 @@ func (hsm *hostStateMachine) checkDelayedHost(info *reconcileInfo) actionResult
 
 	// Check if there's a free slot for hosts that have been previously delayed
 	if info.host.Status.OperationalStatus == metal3v1alpha1.OperationalStatusDelayed {
-		if actionRes := hsm.ensureProvisioningCapacity(info); actionRes != nil {
+		if actionRes := hsm.ensureCapacity(info, info.host.Status.Provisioning.State); actionRes != nil {
 			return actionRes
 		}
 
@@ -156,8 +158,9 @@ func (hsm *hostStateMachine) checkDelayedHost(info *reconcileInfo) actionResult
 	// Make sure the check is re-applied when provisioning an
 	// host not yet tracked by the provisioner
 	switch info.host.Status.Provisioning.State {
-	case metal3v1alpha1.StateInspecting, metal3v1alpha1.StateProvisioning:
-		if actionRes := hsm.ensureProvisioningCapacity(info); actionRes != nil {
+	case metal3v1alpha1.StateInspecting, metal3v1alpha1.StateProvisioning,
+		metal3v1alpha1.StateDeprovisioning, metal3v1alpha1.StateDeleting:
+		if actionRes := hsm.ensureCapacity(info, info.host.Status.Provisioning.State); actionRes != nil {
 			return actionRes
 		}
 	}
diff --git a/controllers/metal3.io/metrics.go b/controllers/metal3.io/metrics.go
index b016f2e6..3113603c 100644
--- a/controllers/metal3.io/metrics.go
+++ b/controllers/metal3.io/metrics.go
@@ -65,6 +65,10 @@ var delayedProvisioningHostCounters = prometheus.NewCounterVec(prometheus.Counte
 	Name: "metal3_delayed__provisioning_total",
 	Help: "The number of times hosts have been delayed while provisioning due a busy provisioner",
 }, []string{labelHostNamespace, labelHostName})
+var delayedDeprovisioningHostCounters = prometheus.NewCounterVec(prometheus.CounterOpts{
+	Name: "metal3_delayed__deprovisioning_total",
+	Help: "The number of times hosts have been delayed while deprovisioning due a busy provisioner",
+}, []string{labelHostNamespace, labelHostName})
 
 var slowOperationBuckets = []float64{30, 90, 180, 360, 720, 1440}
 
@@ -116,7 +120,8 @@ func init() {
 		reconcileErrorCounter,
 		actionFailureCounters,
 		powerChangeAttempts,
-		delayedProvisioningHostCounters)
+		delayedProvisioningHostCounters,
+		delayedDeprovisioningHostCounters)
 
 	for _, collector := range stateTime {
 		metrics.Registry.MustRegister(collector)
diff --git a/docs/configuration.md b/docs/configuration.md
index a958bd98..bb9986f7 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -24,7 +24,7 @@ validation. It is highly recommend to not set it to True.
 `BMO_CONCURRENCY` -- The number of concurrent reconciles performed by the
 Operator. Default is 3.
 
-`PROVISIONING_LIMIT` -- The desired maximum number of hosts that could be provisioned
+`PROVISIONING_LIMIT` -- The desired maximum number of hosts that could be (de)provisioned
 simultaneously by the Operator. The Operator will try to enforce this limit,
 but overflows could happen in case of slow provisioners and / or higher number of
 concurrent reconciles. For such reasons, it is highly recommended to keep
diff --git a/pkg/provisioner/demo/demo.go b/pkg/provisioner/demo/demo.go
index 6714f47d..6ffcc0f8 100644
--- a/pkg/provisioner/demo/demo.go
+++ b/pkg/provisioner/demo/demo.go
@@ -77,7 +77,7 @@ func New(hostData provisioner.HostData, publisher provisioner.EventPublisher) (p
 	return p, nil
 }
 
-func (m *demoProvisioner) HasProvisioningCapacity() (result bool, err error) {
+func (m *demoProvisioner) HasCapacity() (result bool, err error) {
 	return true, nil
 }
 
diff --git a/pkg/provisioner/fixture/fixture.go b/pkg/provisioner/fixture/fixture.go
index 1bbaf378..2b2b6a62 100644
--- a/pkg/provisioner/fixture/fixture.go
+++ b/pkg/provisioner/fixture/fixture.go
@@ -89,7 +89,7 @@ func (f *Fixture) SetValidateError(message string) {
 	f.validateError = message
 }
 
-func (p *fixtureProvisioner) HasProvisioningCapacity() (result bool, err error) {
+func (p *fixtureProvisioner) HasCapacity() (result bool, err error) {
 	return true, nil
 }
 
diff --git a/pkg/provisioner/ironic/ironic.go b/pkg/provisioner/ironic/ironic.go
index c1c5086a..a9db37de 100644
--- a/pkg/provisioner/ironic/ironic.go
+++ b/pkg/provisioner/ironic/ironic.go
@@ -41,7 +41,7 @@ var (
 	ironicInsecure            bool
 	ironicAuth                clients.AuthConfig
 	inspectorAuth             clients.AuthConfig
-	maxProvisioningHosts      int = 20
+	maxBusyHosts              int = 20
 
 	// Keep pointers to ironic and inspector clients configured with
 	// the global auth settings to reuse the connection between
@@ -125,7 +125,7 @@ func init() {
 			fmt.Fprintf(os.Stderr, "Cannot start: Invalid value set for variable PROVISIONING_LIMIT=%s", maxHostsStr)
 			os.Exit(1)
 		}
-		maxProvisioningHosts = value
+		maxBusyHosts = value
 	}
 }
 
@@ -1844,11 +1844,11 @@ func (p *ironicProvisioner) IsReady() (result bool, err error) {
 	return checker.IsReady()
 }
 
-func (p *ironicProvisioner) HasProvisioningCapacity() (result bool, err error) {
+func (p *ironicProvisioner) HasCapacity() (result bool, err error) {
 
-	hosts, err := p.loadProvisioningHosts()
+	hosts, err := p.loadBusyHosts()
 	if err != nil {
-		p.log.Error(err, "Unable to get hosts currently being provisioned")
+		p.log.Error(err, "Unable to get hosts for determining current provisioner capacity")
 		return false, err
 	}
 
@@ -1857,10 +1857,10 @@ func (p *ironicProvisioner) HasProvisioningCapacity() (result bool, err error) {
 		return true, nil
 	}
 
-	return len(hosts) < maxProvisioningHosts, nil
+	return len(hosts) < maxBusyHosts, nil
 }
 
-func (p *ironicProvisioner) loadProvisioningHosts() (hosts map[string]struct{}, err error) {
+func (p *ironicProvisioner) loadBusyHosts() (hosts map[string]struct{}, err error) {
 
 	hosts = make(map[string]struct{})
 	pager := nodes.List(p.client, nodes.ListOpts{
@@ -1885,7 +1885,8 @@ func (p *ironicProvisioner) loadProvisioningHosts() (hosts map[string]struct{},
 		switch nodes.ProvisionState(node.ProvisionState) {
 		case nodes.Cleaning, nodes.CleanWait,
 			nodes.Inspecting, nodes.InspectWait,
-			nodes.Deploying, nodes.DeployWait:
+			nodes.Deploying, nodes.DeployWait,
+			nodes.Deleting:
 			hosts[node.Name] = struct{}{}
 		}
 	}
diff --git a/pkg/provisioner/provisioner.go b/pkg/provisioner/provisioner.go
index 0e70e821..ff593cde 100644
--- a/pkg/provisioner/provisioner.go
+++ b/pkg/provisioner/provisioner.go
@@ -150,8 +150,8 @@ type Provisioner interface {
 	// all the incoming requests.
 	IsReady() (result bool, err error)
 
-	// HasProvisioningCapacity checks if the backend has a free provisioning slot for the current host
-	HasProvisioningCapacity() (result bool, err error)
+	// HasCapacity checks if the backend has a free (de)provisioning slot for the current host
+	HasCapacity() (result bool, err error)
 }
 
 // Result holds the response from a call in the Provsioner API.


Test patch:
diff --git a/controllers/metal3.io/host_state_machine_test.go b/controllers/metal3.io/host_state_machine_test.go
index 595b23b0..e597de48 100644
--- a/controllers/metal3.io/host_state_machine_test.go
+++ b/controllers/metal3.io/host_state_machine_test.go
@@ -123,7 +123,7 @@ func TestProvisioningCapacity(t *testing.T) {
 	for _, tc := range testCases {
 		t.Run(tc.Scenario, func(t *testing.T) {
 			prov := newMockProvisioner()
-			prov.setHasProvisioningCapacity(tc.HasProvisioningCapacity)
+			prov.setHasCapacity(tc.HasProvisioningCapacity)
 			hsm := newHostStateMachine(tc.Host, &BareMetalHostReconciler{}, prov, true)
 			info := makeDefaultReconcileInfo(tc.Host)
 			delayedProvisioningHostCounters.Reset()
@@ -146,6 +146,60 @@ func TestProvisioningCapacity(t *testing.T) {
 	}
 }
 
+func TestDeprovisioningCapacity(t *testing.T) {
+	testCases := []struct {
+		Scenario string
+
+		HasDeprovisioningCapacity bool
+		Host                      *metal3v1alpha1.BareMetalHost
+
+		ExpectedDeprovisioningState metal3v1alpha1.ProvisioningState
+		ExpectedDelayed             bool
+	}{
+		{
+			Scenario:                  "transition-to-deprovisionig-ready",
+			Host:                      host(metal3v1alpha1.StateDeprovisioning).build(),
+			HasDeprovisioningCapacity: true,
+
+			ExpectedDeprovisioningState: metal3v1alpha1.StateReady,
+			ExpectedDelayed:             false,
+		},
+		{
+			Scenario:                  "transition-to-deleting",
+			Host:                      host(metal3v1alpha1.StateDeprovisioning).setDeletion().build(),
+			HasDeprovisioningCapacity: true,
+
+			ExpectedDeprovisioningState: metal3v1alpha1.StateDeleting,
+			ExpectedDelayed:             false,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.Scenario, func(t *testing.T) {
+			prov := newMockProvisioner()
+			prov.setHasCapacity(tc.HasDeprovisioningCapacity)
+			hsm := newHostStateMachine(tc.Host, &BareMetalHostReconciler{}, prov, true)
+			info := makeDefaultReconcileInfo(tc.Host)
+			delayedDeprovisioningHostCounters.Reset()
+
+			result := hsm.ReconcileState(info)
+
+			assert.Equal(t, tc.ExpectedDeprovisioningState, tc.Host.Status.Provisioning.State)
+			assert.Equal(t, tc.ExpectedDelayed, metal3v1alpha1.OperationalStatusDelayed == tc.Host.Status.OperationalStatus, "Expected OperationalStatusDelayed")
+			assert.Equal(t, tc.ExpectedDelayed, assert.ObjectsAreEqual(actionDelayed{}, result), "Expected actionDelayed")
+
+			if tc.ExpectedDelayed {
+				counter, _ := delayedDeprovisioningHostCounters.GetMetricWith(hostMetricLabels(info.request))
+				initialCounterValue := promutil.ToFloat64(counter)
+				for _, sb := range info.postSaveCallbacks {
+					sb()
+				}
+				assert.Greater(t, promutil.ToFloat64(counter), initialCounterValue)
+			}
+		})
+	}
+}
+
 func TestDetach(t *testing.T) {
 	testCases := []struct {
 		Scenario                  string
@@ -817,16 +871,16 @@ func makeDefaultReconcileInfo(host *metal3v1alpha1.BareMetalHost) *reconcileInfo
 
 func newMockProvisioner() *mockProvisioner {
 	return &mockProvisioner{
-		hasProvisioningCapacity: true,
-		nextResults:             make(map[string]provisioner.Result),
-		callsNoError:            make(map[string]bool),
+		hasCapacity:  true,
+		nextResults:  make(map[string]provisioner.Result),
+		callsNoError: make(map[string]bool),
 	}
 }
 
 type mockProvisioner struct {
-	hasProvisioningCapacity bool
-	nextResults             map[string]provisioner.Result
-	callsNoError            map[string]bool
+	hasCapacity  bool
+	nextResults  map[string]provisioner.Result
+	callsNoError map[string]bool
 }
 
 func (m *mockProvisioner) getNextResultByMethod(name string) (result provisioner.Result) {
@@ -838,12 +892,12 @@ func (m *mockProvisioner) getNextResultByMethod(name string) (result provisioner
 	return
 }
 
-func (m *mockProvisioner) setHasProvisioningCapacity(hasCapacity bool) {
-	m.hasProvisioningCapacity = hasCapacity
+func (m *mockProvisioner) setHasCapacity(hasCapacity bool) {
+	m.hasCapacity = hasCapacity
 }
 
-func (m *mockProvisioner) HasProvisioningCapacity() (result bool, err error) {
-	return m.hasProvisioningCapacity, nil
+func (m *mockProvisioner) HasCapacity() (result bool, err error) {
+	return m.hasCapacity, nil
 }
 
 func (m *mockProvisioner) setNextError(methodName, msg string) {
diff --git a/pkg/provisioner/ironic/provisioncapacity_test.go b/pkg/provisioner/ironic/provisioncapacity_test.go
index 1b54c124..a1c9a8ce 100644
--- a/pkg/provisioner/ironic/provisioncapacity_test.go
+++ b/pkg/provisioner/ironic/provisioncapacity_test.go
@@ -12,9 +12,12 @@ import (
 	"github.com/metal3-io/baremetal-operator/pkg/provisioner/ironic/testserver"
 )
 
-func TestHasProvisioningCapacity(t *testing.T) {
+func TestHasCapacity(t *testing.T) {
 
-	provisioningStates := []nodes.ProvisionState{nodes.Cleaning, nodes.CleanWait, nodes.Inspecting, nodes.InspectWait, nodes.Deploying, nodes.DeployWait}
+	states := []nodes.ProvisionState{
+		nodes.Cleaning, nodes.CleanWait, nodes.Inspecting, nodes.InspectWait, nodes.Deploying, nodes.DeployWait,
+		nodes.Deleting,
+	}
 
 	cases := []struct {
 		name              string
@@ -27,22 +30,22 @@ func TestHasProvisioningCapacity(t *testing.T) {
 	}{
 		{
 			name:              "no-capacity",
-			provisioningLimit: 6,
-			nodeStates:        provisioningStates,
+			provisioningLimit: len(states),
+			nodeStates:        states,
 
 			expectedHasCapacity: false,
 		},
 		{
 			name:              "enough-capacity",
-			provisioningLimit: 7,
-			nodeStates:        provisioningStates,
+			provisioningLimit: len(states) + 1,
+			nodeStates:        states,
 
 			expectedHasCapacity: true,
 		},
 		{
 			name:              "ignore-check-if-already-provisioning",
-			provisioningLimit: 6,
-			nodeStates:        provisioningStates,
+			provisioningLimit: len(states),
+			nodeStates:        states,
 			hostName:          "node-1",
 
 			expectedHasCapacity: true,
@@ -78,7 +81,7 @@ func TestHasProvisioningCapacity(t *testing.T) {
 
 			auth := clients.AuthConfig{Type: clients.NoAuth}
 
-			maxProvisioningHosts = tc.provisioningLimit
+			maxBusyHosts = tc.provisioningLimit
 
 			prov, err := newProvisionerWithSettings(host, bmc.Credentials{}, nullEventPublisher,
 				ironic.Endpoint(), auth, inspector.Endpoint(), auth,
@@ -87,7 +90,7 @@ func TestHasProvisioningCapacity(t *testing.T) {
 				t.Fatalf("could not create provisioner: %s", err)
 			}
 
-			result, err := prov.HasProvisioningCapacity()
+			result, err := prov.HasCapacity()
 
 			assert.Equal(t, tc.expectedHasCapacity, result)