Repository: metal3-io/baremetal-operator
Pull request #725 (2020-11-19 16:58:28)

Limit the number of hosts simultaneously provisioned
This PR introduces a soft limit for the number of hosts being simultaneously being provisioned.

Design doc: https://github.com/metal3-io/metal3-docs/blob/master/design/baremetal-operator/limit-hosts-provisioning.md. 

A new configuration variable is provided, called `PROVISIONING_LIMIT` (defaulted to 20), to specify the desired batch size that the operator will try to honor while provisioning new hosts. If there are no available slots, the current action for an host will be rescheduled using a fixed backoff with jittering to spread evenly the delayed requests.
Due the asynchronous behaviour of the Ironic subsystem, and the configurable level of concurrency of the operator, it's not possible to guarantee that such limit will be always respected, and occasional overflows could happen especially during the initial burst; nevertheless the operator will try to enforce such limit as much as possible, and for such reason it's recommended to keep a configuration where PROVISIONING_LIMIT is greater than BMO_CONCURRENCY.

----
__Demo__

Here's a quick demo of the current implementation, using a limit set to 8 and a BMO concurrency level of 3 (demo speeded up). A batch of additional 15 nodes is added to the cluster. It is possible to note the overflow on the initial burst, then the limit is maintained until the provisioning was completed.

[![asciicast](https://asciinema.org/a/9baJxFlOqEh5XlRXCYHrPKQfC.svg)](https://asciinema.org/a/9baJxFlOqEh5XlRXCYHrPKQfC)

metal3-io-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/725#" title="Author self-approved">andfasano</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **russellb**
You can assign the PR to them by writing `/assign @russellb` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["russellb"]} -->
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
golint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334842028294410241) | `/test golint`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
golint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334842028294410241) | `/test golint`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
golint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334842028294410241) | `/test golint`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
golint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334842028294410241) | `/test golint`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
golint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334842028294410241) | `/test golint`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`
gomod | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334842028298604544) | `/test gomod`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
golint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334842028294410241) | `/test golint`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`
gomod | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334842028298604544) | `/test gomod`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
golint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334842028294410241) | `/test golint`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`
gomod | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334842028298604544) | `/test gomod`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`
gofmt | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334842028294410247) | `/test gofmt`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
golint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334842028294410241) | `/test golint`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`
gomod | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334842028298604544) | `/test gomod`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`
gofmt | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334842028294410247) | `/test gofmt`
markdownlint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334842028294410246) | `/test markdownlint`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`
gomod | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334842028298604544) | `/test gomod`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`
gofmt | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334842028294410247) | `/test gofmt`
markdownlint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334842028294410246) | `/test markdownlint`
golint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334843915580215296) | `/test golint`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
unit | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334842028294410243) | `/test unit`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`
gofmt | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334842028294410247) | `/test gofmt`
markdownlint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334842028294410246) | `/test markdownlint`
golint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334843915580215296) | `/test golint`
gomod | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334843915580215297) | `/test gomod`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
govet | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334842028294410240) | `/test govet`
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`
gofmt | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334842028294410247) | `/test gofmt`
markdownlint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334842028294410246) | `/test markdownlint`
golint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334843915580215296) | `/test golint`
gomod | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334843915580215297) | `/test gomod`
unit | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334843915580215298) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
shellcheck | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334842028294410244) | `/test shellcheck`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`
gofmt | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334842028294410247) | `/test gofmt`
markdownlint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334842028294410246) | `/test markdownlint`
golint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334843915580215296) | `/test golint`
gomod | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334843915580215297) | `/test gomod`
unit | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334843915580215298) | `/test unit`
govet | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334843915580215299) | `/test govet`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`
gofmt | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334842028294410247) | `/test gofmt`
markdownlint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334842028294410246) | `/test markdownlint`
golint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334843915580215296) | `/test golint`
gomod | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334843915580215297) | `/test gomod`
unit | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334843915580215298) | `/test unit`
govet | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334843915580215299) | `/test govet`
shellcheck | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334843915580215301) | `/test shellcheck`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
gosec | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334842028294410245) | `/test gosec`
markdownlint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334842028294410246) | `/test markdownlint`
golint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334843915580215296) | `/test golint`
gomod | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334843915580215297) | `/test gomod`
unit | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334843915580215298) | `/test unit`
govet | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334843915580215299) | `/test govet`
shellcheck | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334843915580215301) | `/test shellcheck`
gofmt | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334843915580215303) | `/test gofmt`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
markdownlint | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334842028294410246) | `/test markdownlint`
golint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334843915580215296) | `/test golint`
gomod | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334843915580215297) | `/test gomod`
unit | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334843915580215298) | `/test unit`
govet | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334843915580215299) | `/test govet`
shellcheck | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334843915580215301) | `/test shellcheck`
gofmt | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334843915580215303) | `/test gofmt`
gosec | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334843915580215300) | `/test gosec`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | 2736af0afb47bd90dab12038d99c0cb81fef69ff | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334842028294410242) | `/test generate`
golint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/golint/1334843915580215296) | `/test golint`
gomod | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gomod/1334843915580215297) | `/test gomod`
unit | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/unit/1334843915580215298) | `/test unit`
govet | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/govet/1334843915580215299) | `/test govet`
shellcheck | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/shellcheck/1334843915580215301) | `/test shellcheck`
gofmt | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gofmt/1334843915580215303) | `/test gofmt`
gosec | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/gosec/1334843915580215300) | `/test gosec`
markdownlint | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1334843915580215302) | `/test markdownlint`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
generate | 63bf033a1cd984133df2abcd838047642af7eed8 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/generate/1334843915580215304) | `/test generate`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
markdownlint | 3c4bcc70b88c73941c1c23769376702744de2994 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/725/markdownlint/1335909819877429255) | `/test markdownlint`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/725#" title="Author self-approved">andfasano</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/725#pullrequestreview-551690788" title="Approved">dtantsur</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **russellb**
You can assign the PR to them by writing `/assign @russellb` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["russellb"]} -->
andfasano: > 
> The following combinations of current and target node provisioning states correspond to Hosts in the Provisioning state with what I'd expect (with one exception) to be a fairly high degree of consistency:
> 
>     * `Manageable` -> `TargetProvide`
> 
>     * `Cleaning` -> `TargetProvide`
> 
>     * `CleanWait` -> `TargetProvide`
> 
>     * `CleanFail` (could actually be deprovisioning, but it's brief enough that we can live with it)
> 
>     * `Available` -> `TargetActive`
> 
>     * `Deploying`
> 
>     * `DeployWait`
> 
Apart the `manageable` and `available` states that cannot be used as per the @dtantsur comment, the others have been already in use - the overall goal was to roughly define how busy is currently Ironic (in the initial proposal the subset was slightly larger, including the rescuing/unrescuing/deleting states, which probably could make sense to reintroduce now).

> 
> However, there is a large hole there - once the state becomes `Available` but before we request `TargetActive`, we cannot tell from the ironic states alone whether the node is Provisioning or Ready after deprovisioning. (Unless, of course, we were to store this information somewhere in ironic - which is totally possible.) While setting up the data and requesting a move to active is relatively quick, it may take some time for the operator to notice that the node has reached `Available` and in that window it may start allowing other Hosts to start provisioning.
> 
> If we implement [my proposal](https://groups.google.com/g/metal3-dev/c/N-EpqQn94bw) to add a `Preparing` state _before_ `Ready`, so that the request for TargetActive always happens right at the start of Provisioning, then I think that gating on state changes only would become viable again. Since we want that state anyway, it's worth at least considering whether our path would look easier if we did that first or merged this and then reworked it later, assuming we prefer to gate only on state changes.
> 
Currently gating on state changes seems to work reasonably well, without affecting the convergence of the overall provisioning, so I'd prefer to proceed and merge with this approach and review it later once the new `Preparing` state will be available.

> Use of the error message and error type fields is effective (operational status would be another option, but this is more visible). However, after having written #749 I now think it's a mistake to increment the error count even once because it mixes these routine delays with other possible causes for error. I think all we really need here is a random backoff from a fixed interval. It's fine to add a new actionResult type for this - we actually need one for retry-after-delay anyway.

The ProvisionerBusy error is now used to detect that an host have been previously delayed, so an initial check can be performed even before applying the state machine. If we want to move to a plain action result for the backoff - and also see the usefulness of that - then I suppose operational status could be used to track this situation

andfasano: > > Use of the error message and error type fields is effective (operational status would be another option, but this is more visible). However, after having written #749 I now think it's a mistake to increment the error count even once because it mixes these routine delays with other possible causes for error. I think all we really need here is a random backoff from a fixed interval. It's fine to add a new actionResult type for this - we actually need one for retry-after-delay anyway.
> 
> The ProvisionerBusy error is now used to detect that an host have been previously delayed, so an initial check can be performed even before applying the state machine. If we want to move to a plain action result for the backoff - and also see the usefulness of that - then I suppose operational status could be used to track this situation

Update: I've added a new actionDelayed that randomly backoff in [1m, 2m] and switched to use OperationalStatus to track delayed hosts instead of the error type


andfasano: /test-integration
andfasano: /test-integration
andfasano: /test-integration
andfasano: /test-integration
metal3-io-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/725#" title="Author self-approved">andfasano</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/725#pullrequestreview-551690788" title="Approved">dtantsur</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/725#pullrequestreview-567572673" title="Approved">zaneb</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)~~ [zaneb]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
andfasano: Commits squashed and demo cast updated
andfasano: /test-integration

Patch:
diff --git a/apis/metal3.io/v1alpha1/baremetalhost_types.go b/apis/metal3.io/v1alpha1/baremetalhost_types.go
index 8865d6f7..a4a8781b 100644
--- a/apis/metal3.io/v1alpha1/baremetalhost_types.go
+++ b/apis/metal3.io/v1alpha1/baremetalhost_types.go
@@ -116,6 +116,8 @@ const (
 	// OperationalStatusError is the status value for when the host
 	// has any sort of error.
 	OperationalStatusError OperationalStatus = "error"
+
+	OperationalStatusDelayed = "delayed"
 )
 
 // ErrorType indicates the class of problem that has caused the Host resource
@@ -523,7 +525,7 @@ type BareMetalHostStatus struct {
 	// after modifying this file
 
 	// OperationalStatus holds the status of the host
-	// +kubebuilder:validation:Enum="";OK;discovered;error
+	// +kubebuilder:validation:Enum="";OK;discovered;error;delayed
 	OperationalStatus OperationalStatus `json:"operationalStatus"`
 
 	// ErrorType indicates the type of failure encountered when the
diff --git a/config/crd/bases/metal3.io_baremetalhosts.yaml b/config/crd/bases/metal3.io_baremetalhosts.yaml
index 2ef56a59..32906593 100644
--- a/config/crd/bases/metal3.io_baremetalhosts.yaml
+++ b/config/crd/bases/metal3.io_baremetalhosts.yaml
@@ -525,6 +525,7 @@ spec:
                 - OK
                 - discovered
                 - error
+                - delayed
                 type: string
               poweredOn:
                 description: indicator for whether or not the host is powered on
diff --git a/config/render/capm3.yaml b/config/render/capm3.yaml
index 20ceb5dc..cec08fa7 100644
--- a/config/render/capm3.yaml
+++ b/config/render/capm3.yaml
@@ -523,6 +523,7 @@ spec:
                 - OK
                 - discovered
                 - error
+                - delayed
                 type: string
               poweredOn:
                 description: indicator for whether or not the host is powered on
diff --git a/controllers/metal3.io/action_result.go b/controllers/metal3.io/action_result.go
index a746f05c..0e628c2d 100644
--- a/controllers/metal3.io/action_result.go
+++ b/controllers/metal3.io/action_result.go
@@ -12,7 +12,9 @@ import (
 	"github.com/metal3-io/baremetal-operator/pkg/provisioner"
 )
 
-const maxBackOffCount = 10
+// This is an upper limit for the ErrorCount, so that the max backoff
+// timeout will not exceed (roughly) 8 hours
+const maxBackOffCount = 9
 
 func init() {
 	rand.Seed(time.Now().UTC().UnixNano())
@@ -54,6 +56,18 @@ func (r actionUpdate) Dirty() bool {
 	return true
 }
 
+// actionDelayed it's the same of an actionUpdate, but the requeue time
+// is calculated using a fixed backoff with jitter
+type actionDelayed struct {
+	actionUpdate
+}
+
+func (r actionDelayed) Result() (result reconcile.Result, err error) {
+	result.RequeueAfter = calculateBackoff(1)
+	result.Requeue = true
+	return
+}
+
 // actionComplete is a result indicating that the current action has completed,
 // and that the resource should transition to the next state.
 type actionComplete struct {
@@ -110,6 +124,16 @@ type actionFailed struct {
 	errorCount int
 }
 
+// Distribution sample for errorCount values:
+// 1  [1m, 2m]
+// 2  [2m, 4m]
+// 3  [4m, 8m]
+// 4  [8m, 16m]
+// 5  [16m, 32m]
+// 6  [32m, 1h4m]
+// 7  [1h4m, 2h8m]
+// 8  [2h8m, 4h16m]
+// 9  [4h16m, 8h32m]
 func calculateBackoff(errorCount int) time.Duration {
 
 	if errorCount > maxBackOffCount {
@@ -119,7 +143,7 @@ func calculateBackoff(errorCount int) time.Duration {
 	base := math.Exp2(float64(errorCount))
 	/* #nosec */
 	backOff := base - (rand.Float64() * base * 0.5)
-	backOffDuration := time.Minute * time.Duration(backOff)
+	backOffDuration := time.Duration(float64(time.Minute) * backOff)
 	return backOffDuration
 }
 
diff --git a/controllers/metal3.io/baremetalhost_controller.go b/controllers/metal3.io/baremetalhost_controller.go
index d5967ce2..67b0b9e6 100644
--- a/controllers/metal3.io/baremetalhost_controller.go
+++ b/controllers/metal3.io/baremetalhost_controller.go
@@ -53,9 +53,6 @@ const (
 	rebootAnnotationPrefix        = "reboot.metal3.io"
 )
 
-func init() {
-}
-
 // BareMetalHostReconciler reconciles a BareMetalHost object
 type BareMetalHostReconciler struct {
 	client.Client
@@ -214,7 +211,6 @@ func (r *BareMetalHostReconciler) Reconcile(request ctrl.Request) (result ctrl.R
 	}
 
 	ready, err := prov.IsReady()
-
 	if err != nil {
 		return ctrl.Result{}, errors.Wrap(err, "failed to check services availability")
 	}
@@ -295,6 +291,15 @@ func recordActionFailure(info *reconcileInfo, errorType metal3v1alpha1.ErrorType
 	return actionFailed{dirty: true, ErrorType: errorType, errorCount: info.host.Status.ErrorCount}
 }
 
+func recordActionDelayed(info *reconcileInfo) actionResult {
+
+	counter := delayedProvisioningHostCounters.With(hostMetricLabels(info.request))
+	info.postSaveCallbacks = append(info.postSaveCallbacks, counter.Inc)
+
+	info.host.SetOperationalStatus(metal3v1alpha1.OperationalStatusDelayed)
+	return actionDelayed{}
+}
+
 func (r *BareMetalHostReconciler) credentialsErrorResult(err error, request ctrl.Request, host *metal3v1alpha1.BareMetalHost) (ctrl.Result, error) {
 	switch err.(type) {
 	// In the event a credential secret is defined, but we cannot find it
diff --git a/controllers/metal3.io/host_state_machine.go b/controllers/metal3.io/host_state_machine.go
index 45ba5bb0..4f9a1c77 100644
--- a/controllers/metal3.io/host_state_machine.go
+++ b/controllers/metal3.io/host_state_machine.go
@@ -6,6 +6,7 @@ import (
 	metal3v1alpha1 "github.com/metal3-io/baremetal-operator/apis/metal3.io/v1alpha1"
 	"github.com/metal3-io/baremetal-operator/pkg/provisioner"
 
+	"github.com/pkg/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
@@ -77,9 +78,33 @@ func recordStateEnd(info *reconcileInfo, host *metal3v1alpha1.BareMetalHost, sta
 	return
 }
 
+func (hsm *hostStateMachine) ensureProvisioningCapacity(info *reconcileInfo) actionResult {
+	hasCapacity, err := hsm.Provisioner.HasProvisioningCapacity()
+	if err != nil {
+		return actionError{errors.Wrap(err, "failed to get hosts currently being provisioned")}
+	}
+	if !hasCapacity {
+		return recordActionDelayed(info)
+	}
+
+	return nil
+}
+
 func (hsm *hostStateMachine) updateHostStateFrom(initialState metal3v1alpha1.ProvisioningState,
-	info *reconcileInfo) {
+	info *reconcileInfo) actionResult {
 	if hsm.NextState != initialState {
+
+		// Check if there is a free slot available when trying to
+		// provision an host - if not the action will be delayed.
+		// The check is limited to only the provisioning states to
+		// avoid putting an excessive pressure on the provisioner
+		switch hsm.NextState {
+		case metal3v1alpha1.StateInspecting, metal3v1alpha1.StateProvisioning:
+			if actionRes := hsm.ensureProvisioningCapacity(info); actionRes != nil {
+				return actionRes
+			}
+		}
+
 		info.log.Info("changing provisioning state",
 			"old", initialState,
 			"new", hsm.NextState)
@@ -109,11 +134,47 @@ func (hsm *hostStateMachine) updateHostStateFrom(initialState metal3v1alpha1.Pro
 			}
 		}
 	}
+
+	return nil
 }
 
-func (hsm *hostStateMachine) ReconcileState(info *reconcileInfo) actionResult {
+func (hsm *hostStateMachine) checkDelayedHost(info *reconcileInfo) actionResult {
+
+	// Check if there's a free slot for hosts that have been previously delayed
+	if info.host.Status.OperationalStatus == metal3v1alpha1.OperationalStatusDelayed {
+		if actionRes := hsm.ensureProvisioningCapacity(info); actionRes != nil {
+			return actionRes
+		}
+
+		// A slot is available, let's cleanup the status and retry
+		clearError(info.host)
+		return actionUpdate{}
+	}
+
+	// Make sure the check is re-applied when provisioning an
+	// host not yet tracked by the provisioner
+	switch info.host.Status.Provisioning.State {
+	case metal3v1alpha1.StateInspecting, metal3v1alpha1.StateProvisioning:
+		if actionRes := hsm.ensureProvisioningCapacity(info); actionRes != nil {
+			return actionRes
+		}
+	}
+
+	return nil
+}
+
+func (hsm *hostStateMachine) ReconcileState(info *reconcileInfo) (actionRes actionResult) {
 	initialState := hsm.Host.Status.Provisioning.State
-	defer hsm.updateHostStateFrom(initialState, info)
+
+	defer func() {
+		if overrideAction := hsm.updateHostStateFrom(initialState, info); overrideAction != nil {
+			actionRes = overrideAction
+		}
+	}()
+
+	if delayedResult := hsm.checkDelayedHost(info); delayedResult != nil {
+		return delayedResult
+	}
 
 	if hsm.checkInitiateDelete() {
 		info.log.Info("Initiating host deletion")
diff --git a/controllers/metal3.io/metrics.go b/controllers/metal3.io/metrics.go
index 9e083878..b016f2e6 100644
--- a/controllers/metal3.io/metrics.go
+++ b/controllers/metal3.io/metrics.go
@@ -61,6 +61,10 @@ var hostConfigDataError = prometheus.NewCounterVec(prometheus.CounterOpts{
 	Name: "metal3_host_config_data_error_total",
 	Help: "Number of times the operator has failed to retrieve host configuration data",
 }, []string{labelHostDataType})
+var delayedProvisioningHostCounters = prometheus.NewCounterVec(prometheus.CounterOpts{
+	Name: "metal3_delayed__provisioning_total",
+	Help: "The number of times hosts have been delayed while provisioning due a busy provisioner",
+}, []string{labelHostNamespace, labelHostName})
 
 var slowOperationBuckets = []float64{30, 90, 180, 360, 720, 1440}
 
@@ -111,7 +115,8 @@ func init() {
 		reconcileCounters,
 		reconcileErrorCounter,
 		actionFailureCounters,
-		powerChangeAttempts)
+		powerChangeAttempts,
+		delayedProvisioningHostCounters)
 
 	for _, collector := range stateTime {
 		metrics.Registry.MustRegister(collector)
diff --git a/docs/configuration.md b/docs/configuration.md
index 94bfacec..a958bd98 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -24,6 +24,12 @@ validation. It is highly recommend to not set it to True.
 `BMO_CONCURRENCY` -- The number of concurrent reconciles performed by the
 Operator. Default is 3.
 
+`PROVISIONING_LIMIT` -- The desired maximum number of hosts that could be provisioned
+simultaneously by the Operator. The Operator will try to enforce this limit,
+but overflows could happen in case of slow provisioners and / or higher number of
+concurrent reconciles. For such reasons, it is highly recommended to keep
+BMO_CONCURRENCY value lower than the requested PROVISIONING_LIMIT. Default is 20.
+
 Kustomization Configuration
 ---------------------------
 
diff --git a/pkg/provisioner/demo/demo.go b/pkg/provisioner/demo/demo.go
index 606bb99a..3d336fa3 100644
--- a/pkg/provisioner/demo/demo.go
+++ b/pkg/provisioner/demo/demo.go
@@ -66,6 +66,10 @@ func New(host metal3v1alpha1.BareMetalHost, bmcCreds bmc.Credentials, publisher
 	return p, nil
 }
 
+func (m *demoProvisioner) HasProvisioningCapacity() (result bool, err error) {
+	return true, nil
+}
+
 // ValidateManagementAccess tests the connection information for the
 // host to verify that the location and credentials work.
 func (p *demoProvisioner) ValidateManagementAccess(credentialsChanged, force bool) (result provisioner.Result, provID string, err error) {
diff --git a/pkg/provisioner/empty/empty.go b/pkg/provisioner/empty/empty.go
index 0fbc4880..e3b0c875 100644
--- a/pkg/provisioner/empty/empty.go
+++ b/pkg/provisioner/empty/empty.go
@@ -83,3 +83,7 @@ func (p *emptyProvisioner) PowerOff() (provisioner.Result, error) {
 func (p *emptyProvisioner) IsReady() (bool, error) {
 	return true, nil
 }
+
+func (p *emptyProvisioner) HasProvisioningCapacity() (result bool, err error) {
+	return true, nil
+}
diff --git a/pkg/provisioner/fixture/fixture.go b/pkg/provisioner/fixture/fixture.go
index 13626438..3fb8f7a2 100644
--- a/pkg/provisioner/fixture/fixture.go
+++ b/pkg/provisioner/fixture/fixture.go
@@ -21,6 +21,7 @@ type fixtureHostConfigData struct {
 	metaData    string
 }
 
+// NewHostConfigData creates new host configuration data
 func NewHostConfigData(userData string, networkData string, metaData string) provisioner.HostConfigData {
 	return &fixtureHostConfigData{
 		userData:    userData,
@@ -81,6 +82,10 @@ func (f *Fixture) New(host metal3v1alpha1.BareMetalHost, bmcCreds bmc.Credential
 	return p, nil
 }
 
+func (p *fixtureProvisioner) HasProvisioningCapacity() (result bool, err error) {
+	return true, nil
+}
+
 // ValidateManagementAccess tests the connection information for the
 // host to verify that the location and credentials work.
 func (p *fixtureProvisioner) ValidateManagementAccess(credentialsChanged, force bool) (result provisioner.Result, provID string, err error) {
diff --git a/pkg/provisioner/ironic/ironic.go b/pkg/provisioner/ironic/ironic.go
index 9e364fcb..709dbdd7 100644
--- a/pkg/provisioner/ironic/ironic.go
+++ b/pkg/provisioner/ironic/ironic.go
@@ -3,6 +3,7 @@ package ironic
 import (
 	"fmt"
 	"os"
+	"strconv"
 	"strings"
 	"time"
 
@@ -39,6 +40,7 @@ var (
 	ironicInsecure            bool
 	ironicAuth                clients.AuthConfig
 	inspectorAuth             clients.AuthConfig
+	maxProvisioningHosts      int = 20
 
 	// Keep pointers to ironic and inspector clients configured with
 	// the global auth settings to reuse the connection between
@@ -99,6 +101,15 @@ func init() {
 	if strings.ToLower(ironicInsecureStr) == "true" {
 		ironicInsecure = true
 	}
+
+	if maxHostsStr := os.Getenv("PROVISIONING_LIMIT"); maxHostsStr != "" {
+		value, err := strconv.Atoi(maxHostsStr)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Cannot start: Invalid value set for variable PROVISIONING_LIMIT=%s", maxHostsStr)
+			os.Exit(1)
+		}
+		maxProvisioningHosts = value
+	}
 }
 
 // Provisioner implements the provisioning.Provisioner interface
@@ -1630,3 +1641,52 @@ func (p *ironicProvisioner) IsReady() (result bool, err error) {
 	checker := newIronicDependenciesChecker(p.client, p.inspector, p.log)
 	return checker.IsReady()
 }
+
+func (p *ironicProvisioner) HasProvisioningCapacity() (result bool, err error) {
+
+	hosts, err := p.loadProvisioningHosts()
+	if err != nil {
+		p.log.Error(err, "Unable to get hosts currently being provisioned")
+		return false, err
+	}
+
+	// If the current host is already under processing then let's skip the test
+	if _, ok := hosts[p.host.Name]; ok {
+		return true, nil
+	}
+
+	return len(hosts) < maxProvisioningHosts, nil
+}
+
+func (p *ironicProvisioner) loadProvisioningHosts() (hosts map[string]struct{}, err error) {
+
+	hosts = make(map[string]struct{})
+	pager := nodes.List(p.client, nodes.ListOpts{
+		Fields: []string{"uuid,name,provision_state,driver_internal_info,target_provision_state"},
+	})
+	if pager.Err != nil {
+		return nil, pager.Err
+	}
+
+	page, err := pager.AllPages()
+	if err != nil {
+		return nil, err
+	}
+
+	allNodes, err := nodes.ExtractNodes(page)
+	if err != nil {
+		return nil, err
+	}
+
+	for _, node := range allNodes {
+
+		switch nodes.ProvisionState(node.ProvisionState) {
+		case nodes.Cleaning, nodes.CleanWait,
+			nodes.Inspecting, nodes.InspectWait,
+			nodes.Deploying, nodes.DeployWait:
+			hosts[node.Name] = struct{}{}
+		}
+	}
+
+	return hosts, nil
+}
diff --git a/pkg/provisioner/provisioner.go b/pkg/provisioner/provisioner.go
index 70c9b380..71a3da03 100644
--- a/pkg/provisioner/provisioner.go
+++ b/pkg/provisioner/provisioner.go
@@ -87,6 +87,9 @@ type Provisioner interface {
 	// IsReady checks if the provisioning backend is available to accept
 	// all the incoming requests.
 	IsReady() (result bool, err error)
+
+	// HasProvisioningCapacity checks if the backend has a free provisioning slot for the current host
+	HasProvisioningCapacity() (result bool, err error)
 }
 
 // Result holds the response from a call in the Provsioner API.


Test patch:
diff --git a/controllers/metal3.io/action_result_test.go b/controllers/metal3.io/action_result_test.go
index cb2fd9dc..770a87ee 100644
--- a/controllers/metal3.io/action_result_test.go
+++ b/controllers/metal3.io/action_result_test.go
@@ -11,7 +11,7 @@ import (
 func TestBackoffIncrements(t *testing.T) {
 
 	var backOff time.Duration
-	for i := 0; i < maxBackOffCount; i++ {
+	for i := 1; i <= maxBackOffCount; i++ {
 		prev := backOff
 		backOff = calculateBackoff(i)
 
diff --git a/controllers/metal3.io/baremetalhost_controller_test.go b/controllers/metal3.io/baremetalhost_controller_test.go
index e00d8e7d..4cff51bf 100644
--- a/controllers/metal3.io/baremetalhost_controller_test.go
+++ b/controllers/metal3.io/baremetalhost_controller_test.go
@@ -1290,12 +1290,14 @@ func TestUpdateEventHandler(t *testing.T) {
 
 func TestErrorCountIncrementsAlways(t *testing.T) {
 
+	errorTypes := []metal3v1alpha1.ErrorType{metal3v1alpha1.RegistrationError, metal3v1alpha1.InspectionError, metal3v1alpha1.ProvisioningError, metal3v1alpha1.PowerManagementError}
+
 	b := &metal3v1alpha1.BareMetalHost{}
 	assert.Equal(t, b.Status.ErrorCount, 0)
 
-	setErrorMessage(b, metal3v1alpha1.RegistrationError, "An error message")
-	assert.Equal(t, b.Status.ErrorCount, 1)
-
-	setErrorMessage(b, metal3v1alpha1.InspectionError, "Another error message")
-	assert.Equal(t, b.Status.ErrorCount, 2)
+	for _, c := range errorTypes {
+		before := b.Status.ErrorCount
+		setErrorMessage(b, c, "An error message")
+		assert.Equal(t, before+1, b.Status.ErrorCount)
+	}
 }
diff --git a/controllers/metal3.io/host_state_machine_test.go b/controllers/metal3.io/host_state_machine_test.go
index 58d9a3d2..39e60d80 100644
--- a/controllers/metal3.io/host_state_machine_test.go
+++ b/controllers/metal3.io/host_state_machine_test.go
@@ -2,6 +2,7 @@ package controllers
 
 import (
 	"testing"
+	"time"
 
 	metal3v1alpha1 "github.com/metal3-io/baremetal-operator/apis/metal3.io/v1alpha1"
 	"github.com/metal3-io/baremetal-operator/pkg/bmc"
@@ -11,6 +12,7 @@ import (
 
 	"github.com/metal3-io/baremetal-operator/apis/metal3.io/v1alpha1"
 	"github.com/metal3-io/baremetal-operator/pkg/provisioner"
+	promutil "github.com/prometheus/client_golang/prometheus/testutil"
 	corev1 "k8s.io/api/core/v1"
 	ctrl "sigs.k8s.io/controller-runtime"
 	logf "sigs.k8s.io/controller-runtime/pkg/log"
@@ -23,6 +25,126 @@ func testStateMachine(host *metal3v1alpha1.BareMetalHost) *hostStateMachine {
 	return newHostStateMachine(host, r, p, true)
 }
 
+func TestProvisioningCapacity(t *testing.T) {
+	testCases := []struct {
+		Scenario string
+
+		HasProvisioningCapacity bool
+		Host                    *metal3v1alpha1.BareMetalHost
+
+		ExpectedProvisioningState metal3v1alpha1.ProvisioningState
+		ExpectedDelayed           bool
+	}{
+		{
+			Scenario:                "transition-to-inspecting-delayed",
+			Host:                    host(metal3v1alpha1.StateRegistering).build(),
+			HasProvisioningCapacity: false,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateRegistering,
+			ExpectedDelayed:           true,
+		},
+		{
+			Scenario:                "transition-to-provisioning-delayed",
+			Host:                    host(metal3v1alpha1.StateReady).build(),
+			HasProvisioningCapacity: false,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateReady,
+			ExpectedDelayed:           true,
+		},
+		{
+			Scenario:                "transition-to-inspecting-ok",
+			Host:                    host(metal3v1alpha1.StateRegistering).build(),
+			HasProvisioningCapacity: true,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateInspecting,
+			ExpectedDelayed:           false,
+		},
+		{
+			Scenario:                "transition-to-provisioning-ok",
+			Host:                    host(metal3v1alpha1.StateReady).build(),
+			HasProvisioningCapacity: true,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateProvisioning,
+			ExpectedDelayed:           false,
+		},
+
+		{
+			Scenario:                "already-delayed-delayed",
+			Host:                    host(metal3v1alpha1.StateReady).SetOperationalStatus(metal3v1alpha1.OperationalStatusDelayed).build(),
+			HasProvisioningCapacity: false,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateReady,
+			ExpectedDelayed:           true,
+		},
+		{
+			Scenario:                "already-delayed-ok",
+			Host:                    host(metal3v1alpha1.StateReady).SetOperationalStatus(metal3v1alpha1.OperationalStatusDelayed).build(),
+			HasProvisioningCapacity: true,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateReady,
+			ExpectedDelayed:           false,
+		},
+
+		{
+			Scenario:                "untracked-inspecting-delayed",
+			Host:                    host(metal3v1alpha1.StateInspecting).build(),
+			HasProvisioningCapacity: false,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateInspecting,
+			ExpectedDelayed:           true,
+		},
+		{
+			Scenario:                "untracked-inspecting-ok",
+			Host:                    host(metal3v1alpha1.StateInspecting).build(),
+			HasProvisioningCapacity: true,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateMatchProfile,
+			ExpectedDelayed:           false,
+		},
+		{
+			Scenario:                "untracked-inspecting-delayed",
+			Host:                    host(metal3v1alpha1.StateProvisioning).build(),
+			HasProvisioningCapacity: false,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateProvisioning,
+			ExpectedDelayed:           true,
+		},
+		{
+			Scenario:                "untracked-provisioning-ok",
+			Host:                    host(metal3v1alpha1.StateProvisioning).build(),
+			HasProvisioningCapacity: true,
+
+			ExpectedProvisioningState: metal3v1alpha1.StateProvisioned,
+			ExpectedDelayed:           false,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.Scenario, func(t *testing.T) {
+			prov := newMockProvisioner()
+			prov.setHasProvisioningCapacity(tc.HasProvisioningCapacity)
+			hsm := newHostStateMachine(tc.Host, &BareMetalHostReconciler{}, prov, true)
+			info := makeDefaultReconcileInfo(tc.Host)
+			delayedProvisioningHostCounters.Reset()
+
+			result := hsm.ReconcileState(info)
+
+			assert.Equal(t, tc.ExpectedProvisioningState, tc.Host.Status.Provisioning.State)
+			assert.Equal(t, tc.ExpectedDelayed, metal3v1alpha1.OperationalStatusDelayed == tc.Host.Status.OperationalStatus, "Expected OperationalStatusDelayed")
+			assert.Equal(t, tc.ExpectedDelayed, assert.ObjectsAreEqual(actionDelayed{}, result), "Expected actionDelayed")
+
+			if tc.ExpectedDelayed {
+				counter, _ := delayedProvisioningHostCounters.GetMetricWith(hostMetricLabels(info.request))
+				initialCounterValue := promutil.ToFloat64(counter)
+				for _, sb := range info.postSaveCallbacks {
+					sb()
+				}
+				assert.Greater(t, promutil.ToFloat64(counter), initialCounterValue)
+			}
+		})
+	}
+}
+
 func TestProvisioningCancelled(t *testing.T) {
 	testCases := []struct {
 		Scenario string
@@ -175,116 +297,119 @@ func TestProvisioningCancelled(t *testing.T) {
 	}
 }
 
-func TestErrorCountIncreasedWhenProvisionerFails(t *testing.T) {
+func TestErrorCountIncreasedOnActionFailure(t *testing.T) {
 
 	tests := []struct {
-		Scenario string
-		Host     *metal3v1alpha1.BareMetalHost
+		Scenario           string
+		Host               *metal3v1alpha1.BareMetalHost
+		ProvisionerErrorOn string
 	}{
 		{
-			Scenario: "inspecting",
-			Host:     host(metal3v1alpha1.StateInspecting).build(),
+			Scenario:           "registration",
+			Host:               host(metal3v1alpha1.StateRegistering).build(),
+			ProvisionerErrorOn: "ValidateManagementAccess",
 		},
 		{
-			Scenario: "ready",
-			Host:     host(metal3v1alpha1.StateReady).build(),
+			Scenario:           "inspecting",
+			Host:               host(metal3v1alpha1.StateInspecting).build(),
+			ProvisionerErrorOn: "InspectHardware",
 		},
 		{
-			Scenario: "deprovisioning",
-			Host:     host(metal3v1alpha1.StateDeprovisioning).build(),
+			Scenario:           "provisioning",
+			Host:               host(metal3v1alpha1.StateProvisioning).SetImageURL("imageSpecUrl").build(),
+			ProvisionerErrorOn: "Provision",
 		},
 		{
-			Scenario: "provisioning",
-			Host:     host(metal3v1alpha1.StateProvisioning).SetImageURL("imageSpecUrl").build(),
+			Scenario:           "deprovisioning",
+			Host:               host(metal3v1alpha1.StateDeprovisioning).build(),
+			ProvisionerErrorOn: "Deprovision",
 		},
 		{
-			Scenario: "externallyProvisioned",
-			Host:     host(metal3v1alpha1.StateExternallyProvisioned).SetExternallyProvisioned().build(),
+			Scenario:           "ready-power-on",
+			Host:               host(metal3v1alpha1.StateReady).SetStatusImageURL("imageSpecUrl").SetStatusPoweredOn(false).build(),
+			ProvisionerErrorOn: "PowerOn",
+		},
+		{
+			Scenario:           "ready-power-off",
+			Host:               host(metal3v1alpha1.StateReady).SetStatusImageURL("imageSpecUrl").SetOnline(false).build(),
+			ProvisionerErrorOn: "PowerOff",
+		},
+		{
+			Scenario:           "externally-provisioned-adopt-failed",
+			Host:               host(metal3v1alpha1.StateExternallyProvisioned).SetExternallyProvisioned().build(),
+			ProvisionerErrorOn: "Adopt",
+		},
+		{
+			Scenario:           "provisioned-adopt-failed",
+			Host:               host(metal3v1alpha1.StateProvisioned).build(),
+			ProvisionerErrorOn: "Adopt",
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.Scenario, func(t *testing.T) {
-			prov := &mockProvisioner{}
+			prov := newMockProvisioner()
 			hsm := newHostStateMachine(tt.Host, &BareMetalHostReconciler{}, prov, true)
 			info := makeDefaultReconcileInfo(tt.Host)
 
-			prov.setNextError("some error")
+			prov.setNextError(tt.ProvisionerErrorOn, "some error")
 			result := hsm.ReconcileState(info)
 
-			assert.Greater(t, tt.Host.Status.ErrorCount, 0)
+			assert.Equal(t, 1, tt.Host.Status.ErrorCount)
 			assert.True(t, result.Dirty())
 		})
 	}
 }
 
-func TestErrorCountIncreasedWhenRegistrationFails(t *testing.T) {
-	bmh := host(metal3v1alpha1.StateRegistering).build()
-	prov := &mockProvisioner{}
-	hsm := newHostStateMachine(bmh, &BareMetalHostReconciler{}, prov, true)
-	info := makeDefaultReconcileInfo(bmh)
-	bmh.Status.GoodCredentials = metal3v1alpha1.CredentialsStatus{}
-
-	prov.setNextError("some error")
-	result := hsm.ReconcileState(info)
-
-	assert.Greater(t, bmh.Status.ErrorCount, 0)
-	assert.True(t, result.Dirty())
-}
-
-func TestErrorCountCleared(t *testing.T) {
+func TestErrorCountClearedOnStateTransition(t *testing.T) {
 
 	tests := []struct {
 		Scenario                     string
 		Host                         *metal3v1alpha1.BareMetalHost
+		TargetState                  metal3v1alpha1.ProvisioningState
 		PreserveErrorCountOnComplete bool
 	}{
 		{
-			Scenario: "registering",
-			Host:     host(metal3v1alpha1.StateRegistering).build(),
+			Scenario:    "registering-to-inspecting",
+			Host:        host(metal3v1alpha1.StateRegistering).build(),
+			TargetState: metal3v1alpha1.StateInspecting,
 		},
 		{
-			Scenario: "inspecting",
-			Host:     host(metal3v1alpha1.StateInspecting).build(),
+			Scenario:    "inspecting-to-matchprofile",
+			Host:        host(metal3v1alpha1.StateInspecting).build(),
+			TargetState: metal3v1alpha1.StateMatchProfile,
 		},
 		{
-			Scenario:                     "ready",
-			Host:                         host(metal3v1alpha1.StateReady).build(),
-			PreserveErrorCountOnComplete: true,
+			Scenario:    "matchprofile-to-ready",
+			Host:        host(metal3v1alpha1.StateMatchProfile).build(),
+			TargetState: metal3v1alpha1.StateReady,
 		},
 		{
-			Scenario: "deprovisioning",
-			Host:     host(metal3v1alpha1.StateDeprovisioning).build(),
+			Scenario:    "provisioning-to-provisioned",
+			Host:        host(metal3v1alpha1.StateProvisioning).build(),
+			TargetState: metal3v1alpha1.StateProvisioned,
 		},
 		{
-			Scenario: "provisioning",
-			Host:     host(metal3v1alpha1.StateProvisioning).SetImageURL("imageSpecUrl").build(),
+			Scenario:    "deprovisioning-to-ready",
+			Host:        host(metal3v1alpha1.StateDeprovisioning).build(),
+			TargetState: metal3v1alpha1.StateReady,
 		},
 		{
-			Scenario:                     "externallyProvisioned",
-			Host:                         host(metal3v1alpha1.StateExternallyProvisioned).SetExternallyProvisioned().build(),
-			PreserveErrorCountOnComplete: true,
+			Scenario:    "deprovisioning-to-deleting",
+			Host:        host(metal3v1alpha1.StateDeprovisioning).setDeletion().build(),
+			TargetState: metal3v1alpha1.StateDeleting,
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.Scenario, func(t *testing.T) {
-			prov := &mockProvisioner{}
+			prov := newMockProvisioner()
 			hsm := newHostStateMachine(tt.Host, &BareMetalHostReconciler{}, prov, true)
 			info := makeDefaultReconcileInfo(tt.Host)
 
 			info.host.Status.ErrorCount = 1
-			prov.setNextResult(true)
-			result := hsm.ReconcileState(info)
-
-			assert.Equal(t, 1, tt.Host.Status.ErrorCount)
-			assert.True(t, result.Dirty())
-
-			prov.setNextResult(false)
 			hsm.ReconcileState(info)
-			if tt.PreserveErrorCountOnComplete {
-				assert.Equal(t, 1, tt.Host.Status.ErrorCount)
-			} else {
-				assert.Equal(t, 0, tt.Host.Status.ErrorCount)
-			}
+
+			assert.Equal(t, tt.TargetState, info.host.Status.Provisioning.State)
+			assert.Equal(t, info.host.Status.ErrorCount, 0)
 		})
 	}
 }
@@ -303,13 +428,6 @@ func TestErrorClean(t *testing.T) {
 				SetStatusError(metal3v1alpha1.OperationalStatusError, metal3v1alpha1.RegistrationError, "some error", 1).
 				build(),
 		},
-		{
-			Scenario: "not-clean-after-provisioned-registration-error",
-			Host: host(metal3v1alpha1.StateInspecting).
-				SetStatusError(metal3v1alpha1.OperationalStatusError, metal3v1alpha1.ProvisionedRegistrationError, "some error", 1).
-				build(),
-			ExpectError: true,
-		},
 		{
 			Scenario: "clean-after-creds-change",
 			Host: host(metal3v1alpha1.StateReady).
@@ -320,7 +438,7 @@ func TestErrorClean(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.Scenario, func(t *testing.T) {
-			prov := &mockProvisioner{}
+			prov := newMockProvisioner()
 			hsm := newHostStateMachine(tt.Host, &BareMetalHostReconciler{}, prov, true)
 
 			info := makeDefaultReconcileInfo(tt.Host)
@@ -331,10 +449,10 @@ func TestErrorClean(t *testing.T) {
 			hsm.ReconcileState(info)
 
 			if tt.ExpectError {
-				assert.Equal(t, tt.Host.Status.ErrorType, v1alpha1.ProvisionedRegistrationError)
+				assert.Equal(t, v1alpha1.ProvisionedRegistrationError, tt.Host.Status.ErrorType)
 				assert.NotEmpty(t, tt.Host.Status.ErrorMessage)
 			} else {
-				assert.Equal(t, tt.Host.Status.OperationalStatus, v1alpha1.OperationalStatusOK)
+				assert.Equal(t, v1alpha1.OperationalStatusOK, tt.Host.Status.OperationalStatus)
 				assert.Empty(t, tt.Host.Status.ErrorType)
 				assert.Empty(t, tt.Host.Status.ErrorMessage)
 			}
@@ -347,20 +465,36 @@ type hostBuilder struct {
 }
 
 func host(state metal3v1alpha1.ProvisioningState) *hostBuilder {
+
+	creds := metal3v1alpha1.CredentialsStatus{
+		Reference: &corev1.SecretReference{
+			Name:      "secretRefName",
+			Namespace: "secretNs",
+		},
+		Version: "100",
+	}
+
 	return &hostBuilder{
 		metal3v1alpha1.BareMetalHost{
+			Spec: v1alpha1.BareMetalHostSpec{
+				Online: true,
+				Image: &v1alpha1.Image{
+					URL: "not-empty",
+				},
+				RootDeviceHints: &v1alpha1.RootDeviceHints{},
+			},
 			Status: metal3v1alpha1.BareMetalHostStatus{
 				Provisioning: metal3v1alpha1.ProvisionStatus{
 					State:    state,
 					BootMode: v1alpha1.DefaultBootMode,
-				},
-				GoodCredentials: metal3v1alpha1.CredentialsStatus{
-					Reference: &corev1.SecretReference{
-						Name:      "secretRefName",
-						Namespace: "secretNs",
+					Image: v1alpha1.Image{
+						URL: "", //needs provisioning
 					},
-					Version: "100",
 				},
+				GoodCredentials:   creds,
+				TriedCredentials:  creds,
+				OperationalStatus: metal3v1alpha1.OperationalStatusOK,
+				PoweredOn:         true,
 			},
 		},
 	}
@@ -396,6 +530,32 @@ func (hb *hostBuilder) SetStatusError(opStatus metal3v1alpha1.OperationalStatus,
 	return hb
 }
 
+func (hb *hostBuilder) SetStatusImageURL(url string) *hostBuilder {
+	hb.Status.Provisioning.Image.URL = url
+	return hb
+}
+
+func (hb *hostBuilder) SetStatusPoweredOn(status bool) *hostBuilder {
+	hb.Status.PoweredOn = status
+	return hb
+}
+
+func (hb *hostBuilder) SetOnline(status bool) *hostBuilder {
+	hb.Spec.Online = status
+	return hb
+}
+
+func (hb *hostBuilder) SetOperationalStatus(status metal3v1alpha1.OperationalStatus) *hostBuilder {
+	hb.Status.OperationalStatus = status
+	return hb
+}
+
+func (hb *hostBuilder) setDeletion() *hostBuilder {
+	date := metav1.Date(2021, time.January, 18, 10, 18, 0, 0, time.UTC)
+	hb.DeletionTimestamp = &date
+	return hb
+}
+
 func makeDefaultReconcileInfo(host *metal3v1alpha1.BareMetalHost) *reconcileInfo {
 	return &reconcileInfo{
 		log:     logf.Log.WithName("controllers").WithName("BareMetalHost").WithName("host_state_machine"),
@@ -411,30 +571,46 @@ func makeDefaultReconcileInfo(host *metal3v1alpha1.BareMetalHost) *reconcileInfo
 	}
 }
 
+func newMockProvisioner() *mockProvisioner {
+	return &mockProvisioner{
+		hasProvisioningCapacity: true,
+		nextResults:             make(map[string]provisioner.Result),
+	}
+}
+
 type mockProvisioner struct {
-	nextResult provisioner.Result
+	hasProvisioningCapacity bool
+	nextResults             map[string]provisioner.Result
 }
 
-func (m *mockProvisioner) setNextError(msg string) {
-	m.nextResult = provisioner.Result{
-		ErrorMessage: msg,
+func (m *mockProvisioner) getNextResultByMethod(name string) (result provisioner.Result) {
+	if value, ok := m.nextResults[name]; ok {
+		result = value
 	}
+	return
+}
+
+func (m *mockProvisioner) setHasProvisioningCapacity(hasCapacity bool) {
+	m.hasProvisioningCapacity = hasCapacity
 }
 
-func (m *mockProvisioner) setNextResult(dirty bool) {
-	m.nextResult = provisioner.Result{
-		Dirty:        dirty,
-		ErrorMessage: "",
+func (m *mockProvisioner) HasProvisioningCapacity() (result bool, err error) {
+	return m.hasProvisioningCapacity, nil
+}
+
+func (m *mockProvisioner) setNextError(methodName, msg string) {
+	m.nextResults[methodName] = provisioner.Result{
+		ErrorMessage: msg,
 	}
 }
 
 func (m *mockProvisioner) ValidateManagementAccess(credentialsChanged, force bool) (result provisioner.Result, provID string, err error) {
-	return m.nextResult, "", err
+	return m.getNextResultByMethod("ValidateManagementAccess"), "", err
 }
 
 func (m *mockProvisioner) InspectHardware(force bool) (result provisioner.Result, details *metal3v1alpha1.HardwareDetails, err error) {
 	details = &metal3v1alpha1.HardwareDetails{}
-	return m.nextResult, details, err
+	return m.getNextResultByMethod("InspectHardware"), details, err
 }
 
 func (m *mockProvisioner) UpdateHardwareState() (hwState provisioner.HardwareState, err error) {
@@ -442,27 +618,27 @@ func (m *mockProvisioner) UpdateHardwareState() (hwState provisioner.HardwareSta
 }
 
 func (m *mockProvisioner) Adopt(force bool) (result provisioner.Result, err error) {
-	return m.nextResult, err
+	return m.getNextResultByMethod("Adopt"), err
 }
 
 func (m *mockProvisioner) Provision(configData provisioner.HostConfigData) (result provisioner.Result, err error) {
-	return m.nextResult, err
+	return m.getNextResultByMethod("Provision"), err
 }
 
 func (m *mockProvisioner) Deprovision(force bool) (result provisioner.Result, err error) {
-	return m.nextResult, err
+	return m.getNextResultByMethod("Deprovision"), err
 }
 
 func (m *mockProvisioner) Delete() (result provisioner.Result, err error) {
-	return m.nextResult, err
+	return m.getNextResultByMethod("Delete"), err
 }
 
 func (m *mockProvisioner) PowerOn() (result provisioner.Result, err error) {
-	return m.nextResult, err
+	return m.getNextResultByMethod("PowerOn"), err
 }
 
 func (m *mockProvisioner) PowerOff() (result provisioner.Result, err error) {
-	return m.nextResult, err
+	return m.getNextResultByMethod("PowerOff"), err
 }
 
 func (m *mockProvisioner) IsReady() (result bool, err error) {
diff --git a/pkg/provisioner/ironic/provisioncapacity_test.go b/pkg/provisioner/ironic/provisioncapacity_test.go
new file mode 100644
index 00000000..f37f0c27
--- /dev/null
+++ b/pkg/provisioner/ironic/provisioncapacity_test.go
@@ -0,0 +1,102 @@
+package ironic
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/gophercloud/gophercloud/openstack/baremetal/v1/nodes"
+	"github.com/stretchr/testify/assert"
+
+	"github.com/metal3-io/baremetal-operator/pkg/bmc"
+	"github.com/metal3-io/baremetal-operator/pkg/provisioner/ironic/clients"
+	"github.com/metal3-io/baremetal-operator/pkg/provisioner/ironic/testserver"
+)
+
+func TestHasProvisioningCapacity(t *testing.T) {
+
+	provisioningStates := []nodes.ProvisionState{nodes.Cleaning, nodes.CleanWait, nodes.Inspecting, nodes.InspectWait, nodes.Deploying, nodes.DeployWait}
+
+	cases := []struct {
+		name              string
+		provisioningLimit int
+		nodeStates        []nodes.ProvisionState
+		hostName          string
+
+		expectedHasCapacity bool
+		expectedError       string
+	}{
+		{
+			name:              "no-capacity",
+			provisioningLimit: 6,
+			nodeStates:        provisioningStates,
+
+			expectedHasCapacity: false,
+		},
+		{
+			name:              "enough-capacity",
+			provisioningLimit: 7,
+			nodeStates:        provisioningStates,
+
+			expectedHasCapacity: true,
+		},
+		{
+			name:              "ignore-check-if-already-provisioning",
+			provisioningLimit: 6,
+			nodeStates:        provisioningStates,
+			hostName:          "node-1",
+
+			expectedHasCapacity: true,
+		},
+		{
+			name:              "enough-capacity-due-not-provisioning-states",
+			provisioningLimit: 1,
+			nodeStates:        []nodes.ProvisionState{nodes.Active, nodes.AdoptFail, nodes.Adopting, nodes.Available, nodes.CleanFail},
+
+			expectedHasCapacity: true,
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+
+			allNodes := []nodes.Node{}
+			for n, state := range tc.nodeStates {
+				allNodes = append(allNodes, nodes.Node{
+					Name:           fmt.Sprintf("node-%d", n),
+					ProvisionState: string(state),
+				})
+			}
+
+			ironic := testserver.NewIronic(t).Nodes(allNodes).Start()
+			defer ironic.Stop()
+
+			inspector := testserver.NewInspector(t).Start()
+			defer inspector.Stop()
+
+			host := makeHost()
+			host.Name = tc.hostName
+
+			auth := clients.AuthConfig{Type: clients.NoAuth}
+
+			maxProvisioningHosts = tc.provisioningLimit
+
+			prov, err := newProvisionerWithSettings(host, bmc.Credentials{}, nullEventPublisher,
+				ironic.Endpoint(), auth, inspector.Endpoint(), auth,
+			)
+			if err != nil {
+				t.Fatalf("could not create provisioner: %s", err)
+			}
+
+			result, err := prov.HasProvisioningCapacity()
+
+			assert.Equal(t, tc.expectedHasCapacity, result)
+
+			if tc.expectedError == "" {
+				assert.NoError(t, err)
+			} else {
+				assert.Error(t, err)
+				assert.Regexp(t, tc.expectedError, err.Error())
+			}
+		})
+	}
+}
diff --git a/pkg/provisioner/ironic/testserver/ironic.go b/pkg/provisioner/ironic/testserver/ironic.go
index e5e825a2..44bd52b2 100644
--- a/pkg/provisioner/ironic/testserver/ironic.go
+++ b/pkg/provisioner/ironic/testserver/ironic.go
@@ -254,3 +254,15 @@ func (m *IronicMock) Port(port ports.Port) *IronicMock {
 
 	return m
 }
+
+// Nodes configure the server with a valid response for /v1/nodes
+func (m *IronicMock) Nodes(allNodes []nodes.Node) *IronicMock {
+	resp := struct {
+		Nodes []nodes.Node `json:"nodes"`
+	}{
+		Nodes: allNodes,
+	}
+
+	m.ResponseJSON(m.buildURL("/v1/nodes", http.MethodGet), resp)
+	return m
+}
diff --git a/pkg/provisioner/ironic/testserver/server.go b/pkg/provisioner/ironic/testserver/server.go
index b66a8de7..4e35f1ee 100644
--- a/pkg/provisioner/ironic/testserver/server.go
+++ b/pkg/provisioner/ironic/testserver/server.go
@@ -70,12 +70,12 @@ func (m *MockServer) Endpoint() string {
 
 func (m *MockServer) logRequest(r *http.Request, response string) {
 	m.t.Logf("%s: %s %s -> %s", m.name, r.Method, r.URL, response)
-	m.Requests += r.RequestURI + ";"
+	m.Requests += r.URL.Path + ";"
 
 	bodyRaw, _ := ioutil.ReadAll(r.Body)
 
 	m.FullRequests = append(m.FullRequests, simpleRequest{
-		pattern: r.URL.String(),
+		pattern: r.URL.Path,
 		method:  r.Method,
 		body:    string(bodyRaw),
 	})
@@ -93,8 +93,7 @@ func (m *MockServer) Handler(pattern string, handlerFunc http.HandlerFunc) *Mock
 func (m *MockServer) buildHandler(pattern string) func(http.ResponseWriter, *http.Request) {
 
 	handler := func(w http.ResponseWriter, r *http.Request) {
-
-		if response, ok := m.responsesByMethod[r.URL.String()][r.Method]; ok {
+		if response, ok := m.responsesByMethod[r.URL.Path][r.Method]; ok {
 			m.sendData(w, r, response.code, response.payload)
 			return
 		}
@@ -224,7 +223,7 @@ func (m *MockServer) AddDefaultResponse(patternWithVars string, httpMethod strin
 
 func (m *MockServer) defaultHandler(w http.ResponseWriter, r *http.Request) {
 
-	url := r.URL.String()
+	url := r.URL.Path
 	method := r.Method
 
 	for _, response := range m.defaultResponses {