Repository: metal3-io/baremetal-operator
Pull request #574 (2020-07-06 11:45:06)

Check provisioner ready status when applying reconcile logic
This PR aims to improve the reconcile loop by checking the status of the Provisioner dependencies before trying to reconcile the state: if the provisioner is not ready, the current request is requeued.
In particular, it checks the availability of the Ironic and Ironic Inspector services (thanks to @stbenjam for specific check logic, extracted from https://github.com/openshift-metal3/terraform-provider-ironic/blob/master/ironic/provider.go).
This approach should help especially during the bootstrap of those deployment scenarios where the Ironic services are launched at the same time of the operator - where the reconcile loop could start to operate before Ironic services were effectively up and running.

metal3-io-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/574#" title="Author self-approved">andfasano</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **dhellmann**
You can assign the PR to them by writing `/assign @dhellmann` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["dhellmann"]} -->
metal3-io-bot: Hi @andfasano. Thanks for your PR.

I'm waiting for a [metal3-io](https://github.com/orgs/metal3-io/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://github.com/orgs/metal3-io/people) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

dhellmann: /ok-to-test
dhellmann: /test-integration
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | cac72af21f21124fd4975e0483ba13bb97eca549 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/574/unit/1280131454071738370) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | cac72af21f21124fd4975e0483ba13bb97eca549 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/574/unit/1280131454071738375) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
operator-sdk-generate | 1c47a5e574d9e2bf89462b644330b5743774a366 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/574/operator-sdk-generate/1280180653345738759) | `/test operator-sdk-generate`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
operator-sdk-generate | 1c47a5e574d9e2bf89462b644330b5743774a366 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/574/operator-sdk-generate/1280180653345738759) | `/test operator-sdk-generate`
govet | 1c47a5e574d9e2bf89462b644330b5743774a366 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/574/govet/1280180653345738752) | `/test govet`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
andfasano: /test-integration
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | fc7b8d2f2f404cce3b45b10bf73ffc25b64dff25 | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/574/unit/1280907482885001218) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | 65d4ce8aba11e6465502dfe8816df51f17df4a0c | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/574/unit/1281134998883143687) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
andfasano: /retest
andfasano: /test-integration
metal3-io-bot: @andfasano: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
unit | 1809bc752c2ca7c11c65178b474ad3acca4662fd | [link](https://prow.apps.ci.metal3.io/view/gcs/metal3-prow-artifacts/pr-logs/pull/metal3-io_baremetal-operator/574/unit/1283080049058123781) | `/test unit`



<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
andfasano: /test unit

metal3-io-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/574#" title="Author self-approved">andfasano</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/574#pullrequestreview-448297608" title="Approved">dhellmann</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)~~ [dhellmann]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
metal3-io-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/metal3-io/baremetal-operator/pull/574#" title="Author self-approved">andfasano</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/574#pullrequestreview-448297608" title="Approved">dhellmann</a>*, *<a href="https://github.com/metal3-io/baremetal-operator/pull/574#pullrequestreview-448323771" title="Approved">zaneb</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=metal3-io%2Fbaremetal-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/metal3-io/baremetal-operator/blob/master/OWNERS)~~ [dhellmann,zaneb]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
dhellmann: /test-integration

This version looks OK to me. I'll leave it open for @zaneb to give the final approval since he also had some comments on earlier drafts.

Patch:
diff --git a/pkg/controller/baremetalhost/baremetalhost_controller.go b/pkg/controller/baremetalhost/baremetalhost_controller.go
index 8433c4b0..4b9630b9 100644
--- a/pkg/controller/baremetalhost/baremetalhost_controller.go
+++ b/pkg/controller/baremetalhost/baremetalhost_controller.go
@@ -40,9 +40,10 @@ import (
 )
 
 const (
-	hostErrorRetryDelay    = time.Second * 10
-	unmanagedRetryDelay    = time.Minute * 10
-	rebootAnnotationPrefix = "reboot.metal3.io"
+	hostErrorRetryDelay           = time.Second * 10
+	unmanagedRetryDelay           = time.Minute * 10
+	provisionerNotReadyRetryDelay = time.Second * 30
+	rebootAnnotationPrefix        = "reboot.metal3.io"
 )
 
 var runInTestMode bool
@@ -288,6 +289,16 @@ func (r *ReconcileBareMetalHost) Reconcile(request reconcile.Request) (result re
 		return reconcile.Result{}, errors.Wrap(err, "failed to create provisioner")
 	}
 
+	ready, err := prov.IsReady()
+
+	if err != nil {
+		return reconcile.Result{}, errors.Wrap(err, "failed to check services availability")
+	}
+	if !ready {
+		reqLogger.Info("provisioner is not ready", "RequeueAfter:", provisionerNotReadyRetryDelay)
+		return reconcile.Result{Requeue: true, RequeueAfter: provisionerNotReadyRetryDelay}, nil
+	}
+
 	stateMachine := newHostStateMachine(host, r, prov)
 	actResult := stateMachine.ReconcileState(info)
 	result, err = actResult.Result()
diff --git a/pkg/provisioner/demo/demo.go b/pkg/provisioner/demo/demo.go
index 37fe5c9f..1374a724 100644
--- a/pkg/provisioner/demo/demo.go
+++ b/pkg/provisioner/demo/demo.go
@@ -303,3 +303,8 @@ func (p *demoProvisioner) PowerOff() (result provisioner.Result, err error) {
 
 	// return result, nil
 }
+
+// IsReady always returns true for the demo provisioner
+func (p *demoProvisioner) IsReady() (result bool, err error) {
+	return true, nil
+}
diff --git a/pkg/provisioner/fixture/fixture.go b/pkg/provisioner/fixture/fixture.go
index 4eb330d8..05093972 100644
--- a/pkg/provisioner/fixture/fixture.go
+++ b/pkg/provisioner/fixture/fixture.go
@@ -15,7 +15,7 @@ var log = logf.Log.WithName("fixture")
 var deprovisionRequeueDelay = time.Second * 10
 var provisionRequeueDelay = time.Second * 10
 
-// Provisioner implements the provisioning.Provisioner interface
+// fixtureProvisioner implements the provisioning.fixtureProvisioner interface
 // and uses Ironic to manage the host.
 type fixtureProvisioner struct {
 	// the host to be managed by this provisioner
@@ -28,15 +28,23 @@ type fixtureProvisioner struct {
 	publisher provisioner.EventPublisher
 	// state to manage the two-step adopt process
 	adopted bool
+	// counter to set the provisioner as ready
+	becomeReadyCounter int
 }
 
-// New returns a new Ironic Provisioner
+// New returns a new Ironic FixtureProvisioner
 func New(host *metal3v1alpha1.BareMetalHost, bmcCreds bmc.Credentials, publisher provisioner.EventPublisher) (provisioner.Provisioner, error) {
+	return NewMock(host, bmcCreds, publisher, 0)
+}
+
+// NewMock is used in tests to build a fixture provisioner and inject additional test parameters
+func NewMock(host *metal3v1alpha1.BareMetalHost, bmcCreds bmc.Credentials, publisher provisioner.EventPublisher, becomeReadyCounter int) (provisioner.Provisioner, error) {
 	p := &fixtureProvisioner{
-		host:      host,
-		bmcCreds:  bmcCreds,
-		log:       log.WithValues("host", host.Name),
-		publisher: publisher,
+		host:               host,
+		bmcCreds:           bmcCreds,
+		log:                log.WithValues("host", host.Name),
+		publisher:          publisher,
+		becomeReadyCounter: becomeReadyCounter,
 	}
 	return p, nil
 }
@@ -236,3 +244,14 @@ func (p *fixtureProvisioner) PowerOff() (result provisioner.Result, err error) {
 
 	return result, nil
 }
+
+// IsReady returns the current availability status of the provisioner
+func (p *fixtureProvisioner) IsReady() (result bool, err error) {
+	p.log.Info("checking provisioner status")
+
+	if p.becomeReadyCounter > 0 {
+		p.becomeReadyCounter--
+	}
+
+	return p.becomeReadyCounter == 0, nil
+}
diff --git a/pkg/provisioner/ironic/dependencies.go b/pkg/provisioner/ironic/dependencies.go
new file mode 100644
index 00000000..84444be1
--- /dev/null
+++ b/pkg/provisioner/ironic/dependencies.go
@@ -0,0 +1,92 @@
+package ironic
+
+import (
+	"strings"
+	"time"
+
+	"github.com/go-logr/logr"
+	"github.com/gophercloud/gophercloud"
+	"github.com/gophercloud/gophercloud/openstack/baremetal/v1/drivers"
+	"github.com/gophercloud/gophercloud/pagination"
+)
+
+const (
+	checkRequeueDelay = time.Second * 10
+)
+
+type ironicDependenciesChecker struct {
+	client    *gophercloud.ServiceClient
+	inspector *gophercloud.ServiceClient
+	log       logr.Logger
+}
+
+func newIronicDependenciesChecker(client *gophercloud.ServiceClient, inspector *gophercloud.ServiceClient, log logr.Logger) *ironicDependenciesChecker {
+	return &ironicDependenciesChecker{
+		client:    client,
+		inspector: inspector,
+		log:       log,
+	}
+}
+
+func (i *ironicDependenciesChecker) IsReady() (result bool, err error) {
+
+	ready, err := i.checkIronic()
+	if ready && err == nil {
+		ready = i.checkIronicInspector()
+	}
+
+	return ready, err
+}
+
+func (i *ironicDependenciesChecker) checkEndpoint(client *gophercloud.ServiceClient) (ready bool) {
+
+	// NOTE: Some versions of Ironic inspector returns 404 for /v1/ but 200 for /v1,
+	// which seems to be the default behavior for Flask. Remove the trailing slash
+	// from the client endpoint.
+	endpoint := strings.TrimSuffix(client.Endpoint, "/")
+
+	_, err := client.Get(endpoint, nil, nil)
+	if err != nil {
+		log.Info("error caught while checking endpoint", "endpoint", client.Endpoint, "error", err)
+	}
+
+	return err == nil
+}
+
+func (i *ironicDependenciesChecker) checkIronic() (ready bool, err error) {
+	ready = i.checkEndpoint(i.client)
+	if ready {
+		ready, err = i.checkIronicConductor()
+	}
+	return ready, err
+}
+
+func (i *ironicDependenciesChecker) checkIronicConductor() (ready bool, err error) {
+
+	pager := drivers.ListDrivers(i.client, drivers.ListDriversOpts{
+		Detail: false,
+	})
+	err = pager.Err
+
+	if err != nil {
+		return ready, err
+	}
+
+	driverCount := 0
+	pager.EachPage(func(page pagination.Page) (bool, error) {
+		actual, driverErr := drivers.ExtractDrivers(page)
+		if driverErr != nil {
+			return false, driverErr
+		}
+		driverCount += len(actual)
+		return true, nil
+	})
+	// If we have any drivers, conductor is up.
+	ready = driverCount > 0
+
+	return ready, err
+}
+
+func (i *ironicDependenciesChecker) checkIronicInspector() (ready bool) {
+	return i.checkEndpoint(i.inspector)
+}
diff --git a/pkg/provisioner/ironic/ironic.go b/pkg/provisioner/ironic/ironic.go
index f678891e..a15e38ae 100644
--- a/pkg/provisioner/ironic/ironic.go
+++ b/pkg/provisioner/ironic/ironic.go
@@ -1394,3 +1394,11 @@ func (p *ironicProvisioner) softPowerOff() (result provisioner.Result, err error
 
 	return result, nil
 }
+
+// IsReady checks if the provisioning backend is available
+func (p *ironicProvisioner) IsReady() (result bool, err error) {
+	p.log.Info("verifying ironic provisioner dependencies")
+
+	checker := newIronicDependenciesChecker(p.client, p.inspector, p.log)
+	return checker.IsReady()
+}
diff --git a/pkg/provisioner/provisioner.go b/pkg/provisioner/provisioner.go
index 55beaf42..34fd7326 100644
--- a/pkg/provisioner/provisioner.go
+++ b/pkg/provisioner/provisioner.go
@@ -83,6 +83,10 @@ type Provisioner interface {
 	// PowerOff ensures the server is powered off independently of any image
 	// provisioning operation.
 	PowerOff() (result Result, err error)
+
+	// IsReady checks if the provisioning backend is available to accept
+	// all the incoming requests.
+	IsReady() (result bool, err error)
 }
 
 // Result holds the response from a call in the Provsioner API.


Test patch:
diff --git a/pkg/controller/baremetalhost/baremetalhost_controller_test.go b/pkg/controller/baremetalhost/baremetalhost_controller_test.go
index e1313560..8caa62a1 100644
--- a/pkg/controller/baremetalhost/baremetalhost_controller_test.go
+++ b/pkg/controller/baremetalhost/baremetalhost_controller_test.go
@@ -23,6 +23,8 @@ import (
 
 	metal3apis "github.com/metal3-io/baremetal-operator/pkg/apis"
 	metal3v1alpha1 "github.com/metal3-io/baremetal-operator/pkg/apis/metal3/v1alpha1"
+	"github.com/metal3-io/baremetal-operator/pkg/bmc"
+	"github.com/metal3-io/baremetal-operator/pkg/provisioner"
 	"github.com/metal3-io/baremetal-operator/pkg/provisioner/fixture"
 	"github.com/metal3-io/baremetal-operator/pkg/utils"
 )
@@ -70,8 +72,7 @@ func newHost(name string, spec *metal3v1alpha1.BareMetalHostSpec) *metal3v1alpha
 		TypeMeta: metav1.TypeMeta{
 			Kind:       "BareMetalHost",
 			APIVersion: "metal3.io/v1alpha1",
-		},
-		ObjectMeta: metav1.ObjectMeta{
+		}, ObjectMeta: metav1.ObjectMeta{
 			Name:      name,
 			Namespace: namespace,
 		},
@@ -94,7 +95,7 @@ func newDefaultHost(t *testing.T) *metal3v1alpha1.BareMetalHost {
 	return newDefaultNamedHost(t.Name(), t)
 }
 
-func newTestReconciler(initObjs ...runtime.Object) *ReconcileBareMetalHost {
+func newTestReconcilerWithProvisionerFactory(factory provisioner.Factory, initObjs ...runtime.Object) *ReconcileBareMetalHost {
 
 	c := fakeclient.NewFakeClient(initObjs...)
 
@@ -105,10 +106,14 @@ func newTestReconciler(initObjs ...runtime.Object) *ReconcileBareMetalHost {
 	return &ReconcileBareMetalHost{
 		client:             c,
 		scheme:             scheme.Scheme,
-		provisionerFactory: fixture.New,
+		provisionerFactory: factory,
 	}
 }
 
+func newTestReconciler(initObjs ...runtime.Object) *ReconcileBareMetalHost {
+	return newTestReconcilerWithProvisionerFactory(fixture.New, initObjs...)
+}
+
 type DoneFunc func(host *metal3v1alpha1.BareMetalHost, result reconcile.Result) bool
 
 func newRequest(host *metal3v1alpha1.BareMetalHost) reconcile.Request {
@@ -1240,3 +1245,26 @@ func TestUpdateBootMode(t *testing.T) {
 		})
 	}
 }
+
+func TestProvisionerIsReady(t *testing.T) {
+	host := newDefaultHost(t)
+
+	var prov provisioner.Provisioner
+	r := newTestReconcilerWithProvisionerFactory(func(host *metal3v1alpha1.BareMetalHost, bmcCreds bmc.Credentials, publisher provisioner.EventPublisher) (provisioner provisioner.Provisioner, err error) {
+		if prov == nil {
+			prov, err = fixture.NewMock(host, bmcCreds, publisher, 5)
+		}
+		return prov, err
+	}, host)
+
+	tryReconcile(t, r, host,
+		func(host *metal3v1alpha1.BareMetalHost, result reconcile.Result) bool {
+			if prov == nil {
+				return false
+			}
+
+			ready, _ := prov.IsReady()
+			return ready
+		},
+	)
+}
diff --git a/pkg/provisioner/ironic/ironic_test.go b/pkg/provisioner/ironic/ironic_test.go
index 8cc022ad..19e19ccd 100644
--- a/pkg/provisioner/ironic/ironic_test.go
+++ b/pkg/provisioner/ironic/ironic_test.go
@@ -2,6 +2,11 @@ package ironic
 
 import (
 	"fmt"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"strconv"
+	"strings"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -20,58 +25,115 @@ func init() {
 	logf.SetLogger(logf.ZapLogger(true))
 }
 
-func TestGetUpdateOptsForNodeWithRootHints(t *testing.T) {
-	rotational := true
-	host := &metal3v1alpha1.BareMetalHost{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      "myhost",
-			Namespace: "myns",
-			UID:       "27720611-e5d1-45d3-ba3a-222dcfaa4ca2",
+func TestProvisionerIsReady(t *testing.T) {
+
+	cases := []struct {
+		name      string
+		ironic    *mockServer
+		inspector *mockServer
+
+		expectedIronicCalls    string
+		expectedInspectorCalls string
+		expectedIsReady        bool
+		expectedError          string
+	}{
+		{
+			name:      "IsReady",
+			ironic:    newMockServer(6385).addDrivers(),
+			inspector: newMockServer(5050),
+
+			expectedIronicCalls:    "localhost:6385/v1;localhost:6385/v1/drivers;",
+			expectedInspectorCalls: "localhost:5050/v1;",
+			expectedIsReady:        true,
 		},
-		Spec: metal3v1alpha1.BareMetalHostSpec{
-			Image: &metal3v1alpha1.Image{
-				URL: "not-empty",
-			},
-			Online:          true,
-			HardwareProfile: "libvirt",
-			RootDeviceHints: &metal3v1alpha1.RootDeviceHints{
-				DeviceName:         "userd_devicename",
-				HCTL:               "1:2:3:4",
-				Model:              "userd_model",
-				Vendor:             "userd_vendor",
-				SerialNumber:       "userd_serial",
-				MinSizeGigabytes:   40,
-				WWN:                "userd_wwn",
-				WWNWithExtension:   "userd_with_extension",
-				WWNVendorExtension: "userd_vendor_extension",
-				Rotational:         &rotational,
-			},
+		{
+			name:      "NoDriversLoaded",
+			ironic:    newMockServer(6385),
+			inspector: newMockServer(5050),
+
+			expectedIronicCalls: "localhost:6385/v1;localhost:6385/v1/drivers;",
 		},
-		Status: metal3v1alpha1.BareMetalHostStatus{
-			Provisioning: metal3v1alpha1.ProvisionStatus{
-				ID: "provisioning-id",
-				// Place the hints in the status field to pretend the
-				// controller has already reconciled partially.
-				RootDeviceHints: &metal3v1alpha1.RootDeviceHints{
-					DeviceName:         "userd_devicename",
-					HCTL:               "1:2:3:4",
-					Model:              "userd_model",
-					Vendor:             "userd_vendor",
-					SerialNumber:       "userd_serial",
-					MinSizeGigabytes:   40,
-					WWN:                "userd_wwn",
-					WWNWithExtension:   "userd_with_extension",
-					WWNVendorExtension: "userd_vendor_extension",
-					Rotational:         &rotational,
-				},
-			},
-			HardwareProfile: "libvirt",
+		{
+			name:      "IronicDown",
+			inspector: newMockServer(5050),
+
+			expectedIsReady: false,
+		},
+		{
+			name:   "InspectorDown",
+			ironic: newMockServer(6385).addDrivers(),
+
+			expectedIronicCalls: "localhost:6385/v1;localhost:6385/v1/drivers;",
+
+			expectedIsReady: false,
+		},
+		{
+			name:      "IronicNotOk",
+			ironic:    newMockServer(6385).setErrorCode(http.StatusInternalServerError),
+			inspector: newMockServer(5050),
+
+			expectedIsReady: false,
+
+			expectedIronicCalls: "localhost:6385/v1;",
+		},
+		{
+			name:      "IronicNotOkAndNotExpected",
+			ironic:    newMockServer(6385).setErrorCode(http.StatusBadGateway),
+			inspector: newMockServer(5050),
+
+			expectedIsReady: false,
+
+			expectedIronicCalls: "localhost:6385/v1;",
+		},
+		{
+			name:      "InspectorNotOk",
+			ironic:    newMockServer(6385).addDrivers(),
+			inspector: newMockServer(5050).setErrorCode(http.StatusInternalServerError),
+
+			expectedIsReady: false,
+
+			expectedIronicCalls:    "localhost:6385/v1;localhost:6385/v1/drivers;",
+			expectedInspectorCalls: "localhost:5050/v1;",
 		},
 	}
 
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if tc.ironic != nil {
+				tc.ironic.start()
+				defer tc.ironic.stop()
+			}
+
+			if tc.inspector != nil {
+				tc.inspector.start()
+				defer tc.inspector.stop()
+			}
+
+			prov, err := newProvisioner(makeHost(), bmc.Credentials{}, nil)
+			ready, err := prov.IsReady()
+
+			if tc.ironic != nil {
+				assert.Equal(t, tc.expectedIronicCalls, tc.ironic.requests)
+			}
+			if tc.inspector != nil {
+				assert.Equal(t, tc.expectedInspectorCalls, tc.inspector.requests)
+			}
+
+			if tc.expectedError != "" {
+				assert.Regexp(t, tc.expectedError, err)
+			} else {
+				assert.Nil(t, err)
+				assert.Equal(t, tc.expectedIsReady, ready)
+			}
+		})
+	}
+}
+
+func TestGetUpdateOptsForNodeWithRootHints(t *testing.T) {
+
 	eventPublisher := func(reason, message string) {}
 
-	prov, err := newProvisioner(host, bmc.Credentials{}, eventPublisher)
+	prov, err := newProvisioner(makeHost(), bmc.Credentials{}, eventPublisher)
 	ironicNode := &nodes.Node{}
 
 	patches, err := prov.getUpdateOptsForNode(ironicNode)
@@ -432,3 +494,129 @@ func TestGetUpdateOptsForNodeBootMode(t *testing.T) {
 		})
 	}
 }
+
+func makeHost() *metal3v1alpha1.BareMetalHost {
+	rotational := true
+
+	return &metal3v1alpha1.BareMetalHost{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "myhost",
+			Namespace: "myns",
+			UID:       "27720611-e5d1-45d3-ba3a-222dcfaa4ca2",
+		},
+		Spec: metal3v1alpha1.BareMetalHostSpec{
+			Image: &metal3v1alpha1.Image{
+				URL: "not-empty",
+			},
+			Online:          true,
+			HardwareProfile: "libvirt",
+			RootDeviceHints: &metal3v1alpha1.RootDeviceHints{
+				DeviceName:         "userd_devicename",
+				HCTL:               "1:2:3:4",
+				Model:              "userd_model",
+				Vendor:             "userd_vendor",
+				SerialNumber:       "userd_serial",
+				MinSizeGigabytes:   40,
+				WWN:                "userd_wwn",
+				WWNWithExtension:   "userd_with_extension",
+				WWNVendorExtension: "userd_vendor_extension",
+				Rotational:         &rotational,
+			},
+		},
+		Status: metal3v1alpha1.BareMetalHostStatus{
+			Provisioning: metal3v1alpha1.ProvisionStatus{
+				ID: "provisioning-id",
+				// Place the hints in the status field to pretend the
+				// controller has already reconciled partially.
+				RootDeviceHints: &metal3v1alpha1.RootDeviceHints{
+					DeviceName:         "userd_devicename",
+					HCTL:               "1:2:3:4",
+					Model:              "userd_model",
+					Vendor:             "userd_vendor",
+					SerialNumber:       "userd_serial",
+					MinSizeGigabytes:   40,
+					WWN:                "userd_wwn",
+					WWNWithExtension:   "userd_with_extension",
+					WWNVendorExtension: "userd_vendor_extension",
+					Rotational:         &rotational,
+				},
+			},
+			HardwareProfile: "libvirt",
+		},
+	}
+}
+
+func newMockServer(port int) *mockServer {
+	return &mockServer{
+		port: strconv.Itoa(port),
+	}
+}
+
+type mockServer struct {
+	port      string
+	requests  string
+	server    *httptest.Server
+	drivers   string
+	errorCode int
+}
+
+func (m *mockServer) setErrorCode(code int) *mockServer {
+	m.errorCode = code
+
+	return m
+}
+
+func (m *mockServer) addDrivers() *mockServer {
+	m.drivers = `
+	{
+		"drivers": [{
+			"hosts": [
+			  "master-2.ostest.test.metalkube.org"
+			],
+			"links": [
+			  {
+				"href": "http://[fd00:1101::3]:6385/v1/drivers/fake-hardware",
+				"rel": "self"
+			  },
+			  {
+				"href": "http://[fd00:1101::3]:6385/drivers/fake-hardware",
+				"rel": "bookmark"
+			  }
+			],
+			"name": "fake-hardware"
+		}]
+	}
+	`
+	return m
+}
+
+func (m *mockServer) start() *mockServer {
+	listener, err := net.Listen("tcp", "127.0.0.1:"+m.port)
+	if err != nil {
+		panic(err)
+	}
+
+	m.server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		m.requests += r.Host + r.RequestURI + ";"
+
+		if m.errorCode != 0 {
+			http.Error(w, "An error", m.errorCode)
+			return
+		}
+
+		if strings.Contains(r.RequestURI, "/v1/drivers") {
+			w.Header().Set("Content-Type", "application/json")
+			fmt.Fprintf(w, m.drivers)
+		}
+	}))
+
+	m.server.Listener.Close()
+	m.server.Listener = listener
+	m.server.Start()
+
+	return m
+}
+
+func (m *mockServer) stop() {
+	m.server.Close()
+}