Repository: gardener/machine-controller-manager (go)

Issue:
Race condition while creating machines in CrashLoopBackoff state
**What happened**:
The orphan controller always used to ignore machines that are being just in creation. I.e that `status.CurrentStatus.Phase` was not set. However, with the introduction of `CrashLoopBackOff`, and when the first try for machine creation fails and any further retries lead to the possibility of VMs showing up on the consoled prior to updating the machine object and hence is treated as an orphan VM. This is causing this race to lead to this state.
Refer - [machine safety code](https://github.com/gardener/machine-controller-manager/blob/master/pkg/controller/machine_safety.go#L575-L580)

**What you expected to happen**:
For the VM to be not treated as an orphan and allow it to join.  

**How to reproduce it (as minimally and precisely as possible)**:

Try creating a VM that should return an error on the first creation and post which the next creations should lead to this race condition on providers that return after too long. 

**Anything else we need to know**:

One way I think of fixing this would be to ignore machines in crashLoopBackoff while trying to clean up orphan machines. 

Sample MCM logs
```
I1215 14:55:12.856005       1 controller_utils.go:599] Controller shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994 created machine shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk
I1215 14:55:12.856529       1 event.go:255] Event(v1.ObjectReference{Kind:"MachineSet", Namespace:"shoot--garden--ccee-m3-eu1", Name:"shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994", UID:"94661b5e-2fac-4006-bc36-6df5a2e0a42b", APIVersion:"machine.sapcloud.io/v1alpha1", ResourceVersion:"463618256", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created Machine: shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk
I1215 14:55:12.873818       1 machine.go:421] Creating machine "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk", please wait!
I1215 14:58:52.954724       1 machine_safety.go:571] SafetyController: Machine object "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk" is being processed by machine controller, hence skipping
I1215 14:59:14.850510       1 machine_safety.go:571] SafetyController: Machine object "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk" is being processed by machine controller, hence skipping
I1215 15:01:38.556883       1 machine_safety.go:571] SafetyController: Machine object "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk" is being processed by machine controller, hence skipping
I1215 15:02:11.576825       1 driver_openstack.go:417] Found machine with name: "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk"
E1215 15:02:15.491244       1 machine.go:478] Error while creating machine shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk: error waiting for the "566dfa8a-088c-480f-a9b3-e8df2f32a8ed" server status: unexpected status "ERROR", wanted target "ACTIVE", fault: {Code:0 Created:0001-01-01 00:00:00 +0000 UTC Details: Message:}
I1215 15:02:15.526334       1 machine.go:421] Creating machine "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk", please wait!
I1215 15:02:25.048400       1 driver_openstack.go:417] Found machine with name: "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk"
I1215 15:02:26.654552       1 driver_openstack.go:417] Found machine with name: "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk"
I1215 15:02:28.529091       1 machine_safety.go:639] SafetyController: Orphan VM found and terminated VM: shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk, openstack:///eu-nl-1/90e0b27a-fe40-43b9-b6ac-ff2e7cc24c21
E1215 15:03:16.153424       1 machine.go:478] Error while creating machine shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk: error waiting for the "90e0b27a-fe40-43b9-b6ac-ff2e7cc24c21" server status: Resource not found
I1215 15:03:16.158002       1 machine.go:843] Not updating the status of the machine object "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk" , as it is already same
I1215 15:04:15.526510       1 machine.go:421] Creating machine "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk", please wait!
I1215 15:11:34.552966       1 driver_openstack.go:417] Found machine with name: "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk"
I1215 15:11:37.942393       1 driver_openstack.go:417] Found machine with name: "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk"
I1215 15:11:39.342763       1 machine_safety.go:639] SafetyController: Orphan VM found and terminated VM: shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk, openstack:///eu-nl-1/8cdfe48d-8f0f-4a9c-ae06-cdcb20dca785
E1215 15:11:41.647753       1 machine.go:478] Error while creating machine shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk: error waiting for the "8cdfe48d-8f0f-4a9c-ae06-cdcb20dca785" server status: Resource not found
I1215 15:11:41.651507       1 machine.go:843] Not updating the status of the machine object "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk" , as it is already same
I1215 15:13:41.657137       1 machine.go:421] Creating machine "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk", please wait!
I1215 15:14:38.246596       1 driver_openstack.go:417] Found machine with name: "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk"
I1215 15:14:39.841766       1 driver_openstack.go:417] Found machine with name: "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk"
I1215 15:14:41.534624       1 machine_safety.go:639] SafetyController: Orphan VM found and terminated VM: shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk, openstack:///eu-nl-1/6e56cc76-5e62-4a32-a099-f233681fe4ba
E1215 15:14:43.053013       1 machine.go:478] Error while creating machine shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk: error waiting for the "6e56cc76-5e62-4a32-a099-f233681fe4ba" server status: Resource not found
I1215 15:14:43.057342       1 machine.go:843] Not updating the status of the machine object "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk" , as it is already same
I1215 15:16:43.064546       1 machine.go:421] Creating machine "shoot--garden--ccee-m3-eu1-cpu-worker-z1-85994-s9kpk", please wait!
```

**Environment**:
/platform openStack
/platform azure 
(i think, this could affect these platforms as they take longer to return on a create call)

cc: @vpnachev 


Pull request:
Avoid the deletion of the machines in CrashLoopBackoff state by the safety controller
**What this PR does / why we need it**:
This PR will avoid the deletion of the machines in CrashLoopBackoff state by the safety controller

**Which issue(s) this PR fixes**:
Fixes #583 

**Special notes for your reviewer**:
More UTs shall be added to test this scenario in the near future. This one is an immediate solution to mitigate the issue.

**Release note**:
<!--  Write your release note:
1. Enter your release note in the below block.
2. If no release note is required, just write "NONE" within the block.

Format of block header: <category> <target_group>
Possible values:
- category:       breaking|feature|bugfix|doc|other
- target_group:   user|operator|developer|dependency
-->
```other operator
Avoid the deletion of the machines in CrashLoopBackoff state by the safety controller
```


Patch:
diff --git a/.docforge/manifest.yaml b/.docforge/manifest.yaml
deleted file mode 100644
index 853e80519..000000000
--- a/.docforge/manifest.yaml
+++ /dev/null
@@ -1,6 +0,0 @@
-nodesSelector:
-  path: https://github.com/gardener/machine-controller-manager/tree/master/docs    
-links:
-  downloads:
-    scope:
-      gardener/machine-controller-manager/(blob|raw)/(.*)/docs: ~
diff --git a/pkg/controller/machine_safety.go b/pkg/controller/machine_safety.go
index b11b6c8c9..d0ea33121 100644
--- a/pkg/controller/machine_safety.go
+++ b/pkg/controller/machine_safety.go
@@ -571,10 +571,9 @@ func (c *controller) checkMachineClass(
 			// Any other types of errors
 			klog.Errorf("SafetyController: Error while trying to GET machines. Error: %s", err)
 		} else if err != nil || machine.Spec.ProviderID != machineID {
-
 			// If machine exists and machine object is still been processed by the machine controller
 			if err == nil &&
-				machine.Status.CurrentStatus.Phase == "" {
+				(machine.Status.CurrentStatus.Phase == "" || machine.Status.CurrentStatus.Phase == v1alpha1.MachineCrashLoopBackOff) {
 				klog.V(3).Infof("SafetyController: Machine object %q is being processed by machine controller, hence skipping", machine.Name)
 				continue
 			}
diff --git a/pkg/driver/driver.go b/pkg/driver/driver.go
index 1bf10ee1f..0d1dbe7a4 100644
--- a/pkg/driver/driver.go
+++ b/pkg/driver/driver.go
@@ -101,16 +101,32 @@ func NewDriver(machineID string, secretData map[string][]byte, classKind string,
 
 	return NewFakeDriver(
 		func() (string, string, error) {
+			fakeVMs["fake"] = "fake_ip"
 			return "fake", "fake_ip", nil
 		},
-		func(string) error {
+		func(machineID string, machineName string) error {
+			fakeVMs[machineID] = machineName
+			return nil
+		},
+		func(machineID string) error {
+			// delete(fakeVMs, "fake")
+			delete(fakeVMs, machineID)
 			return nil
 		},
 		func() (string, error) {
-			return "fake", nil
+			return "", nil
 		},
 		func() (VMs, error) {
-			return nil, nil
+			return fakeVMs, nil
+		},
+		func([]corev1.PersistentVolumeSpec) ([]string, error) {
+			return []string{}, nil
+		},
+		func() string {
+			return ""
+		},
+		func(string) {
+			return
 		},
 	)
 }
diff --git a/pkg/driver/driver_fake.go b/pkg/driver/driver_fake.go
index b3c202101..61710c145 100644
--- a/pkg/driver/driver_fake.go
+++ b/pkg/driver/driver_fake.go
@@ -19,23 +19,41 @@ package driver
 
 import corev1 "k8s.io/api/core/v1"
 
+// fakeVMs keep a list of actively using fake machines for test purposes
+var fakeVMs = make(VMs)
+
 // FakeDriver is a fake driver returned when none of the actual drivers match
 type FakeDriver struct {
-	create func() (string, string, error)
-	delete func(string) error
-	//existing func() (string, v1alpha1.MachinePhase, error)
+	create      func() (string, string, error)
+	add         func(string, string) error
+	delete      func(string) error
 	existing    func() (string, error)
 	getVMs      func() (VMs, error)
 	getVolNames func([]corev1.PersistentVolumeSpec) ([]string, error)
+	getUserData func() string
+	setUserData func(string)
 }
 
 // NewFakeDriver returns a new fakedriver object
-func NewFakeDriver(create func() (string, string, error), delete func(string) error, existing func() (string, error), getVMs func() (VMs, error)) Driver {
+func NewFakeDriver(
+	create func() (string, string, error),
+	add func(string, string) error,
+	delete func(string) error,
+	existing func() (string, error),
+	getVMs func() (VMs, error),
+	getVolNames func([]corev1.PersistentVolumeSpec) ([]string, error),
+	getUserData func() string,
+	setUserData func(string),
+) Driver {
 	return &FakeDriver{
-		create:   create,
-		delete:   delete,
-		existing: existing,
-		getVMs:   getVMs,
+		create:      create,
+		add:         add,
+		delete:      delete,
+		existing:    existing,
+		getVMs:      getVMs,
+		getVolNames: getVolNames,
+		getUserData: getUserData,
+		setUserData: setUserData,
 	}
 }
 
@@ -44,6 +62,11 @@ func (d *FakeDriver) Create() (string, string, error) {
 	return d.create()
 }
 
+// Add add a newly created machine to the fakeVMs list
+func (d *FakeDriver) Add(machineID string, machineName string) error {
+	return d.add(machineID, machineName)
+}
+
 // Delete deletes a fake driver
 func (d *FakeDriver) Delete(machineID string) error {
 	return d.delete(machineID)
diff --git a/pkg/util/provider/driver/fake.go b/pkg/util/provider/driver/fake.go
index 31dd156bf..83b123434 100644
--- a/pkg/util/provider/driver/fake.go
+++ b/pkg/util/provider/driver/fake.go
@@ -25,6 +25,9 @@ import (
 	"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/status"
 )
 
+// VMs is the map to hold the VM data
+type VMs map[string]string
+
 // FakeDriver is a fake driver returned when none of the actual drivers match
 type FakeDriver struct {
 	VMExists       bool
@@ -32,13 +35,31 @@ type FakeDriver struct {
 	NodeName       string
 	LastKnownState string
 	Err            error
+	fakeVMs        VMs
 }
 
 // NewFakeDriver returns a new fakedriver object
-func NewFakeDriver(fakeDriver *FakeDriver) Driver {
+func NewFakeDriver(vmExists bool, providerID, nodeName, lastKnownState string, err error, fakeVMs VMs) Driver {
+	fakeDriver := &FakeDriver{
+		VMExists:       vmExists,
+		ProviderID:     providerID,
+		NodeName:       nodeName,
+		LastKnownState: lastKnownState,
+		Err:            err,
+		fakeVMs:        make(VMs),
+	}
+	if providerID != "" && nodeName != "" {
+		_ = fakeDriver.AddMachine(providerID, nodeName)
+	}
 	return fakeDriver
 }
 
+// AddMachine makes a call to the driver to create the machine.
+func (d *FakeDriver) AddMachine(machineID, machineName string) error {
+	d.fakeVMs[machineID] = machineName
+	return nil
+}
+
 // CreateMachine makes a call to the driver to create the machine.
 func (d *FakeDriver) CreateMachine(ctx context.Context, createMachineRequest *CreateMachineRequest) (*CreateMachineResponse, error) {
 	if d.Err == nil {
@@ -56,6 +77,7 @@ func (d *FakeDriver) CreateMachine(ctx context.Context, createMachineRequest *Cr
 // DeleteMachine make a call to the driver to delete the machine.
 func (d *FakeDriver) DeleteMachine(ctx context.Context, deleteMachineRequest *DeleteMachineRequest) (*DeleteMachineResponse, error) {
 	d.VMExists = false
+	delete(d.fakeVMs, deleteMachineRequest.Machine.Spec.ProviderID)
 	return &DeleteMachineResponse{
 		LastKnownState: d.LastKnownState,
 	}, d.Err
@@ -80,7 +102,7 @@ func (d *FakeDriver) GetMachineStatus(ctx context.Context, getMachineStatusReque
 // ListMachines have to list machines
 func (d *FakeDriver) ListMachines(ctx context.Context, listMachinesRequest *ListMachinesRequest) (*ListMachinesResponse, error) {
 	return &ListMachinesResponse{
-		MachineList: map[string]string{},
+		MachineList: d.fakeVMs,
 	}, d.Err
 }
 
diff --git a/pkg/util/provider/machinecontroller/machine_safety.go b/pkg/util/provider/machinecontroller/machine_safety.go
index d71a20f0e..8f8263f1c 100644
--- a/pkg/util/provider/machinecontroller/machine_safety.go
+++ b/pkg/util/provider/machinecontroller/machine_safety.go
@@ -219,7 +219,7 @@ func (c *controller) checkMachineClass(machineClass *v1alpha1.MachineClass) (mac
 
 			// If machine exists and machine object is still been processed by the machine controller
 			if err == nil &&
-				machine.Status.CurrentStatus.Phase == "" {
+				(machine.Status.CurrentStatus.Phase == "" || machine.Status.CurrentStatus.Phase == v1alpha1.MachineCrashLoopBackOff) {
 				klog.V(3).Infof("SafetyController: Machine object %q is being processed by machine controller, hence skipping", machine.Name)
 				continue
 			}


Test patch:
diff --git a/pkg/controller/machine_safety_test.go b/pkg/controller/machine_safety_test.go
index 25cae16ce..5da122861 100644
--- a/pkg/controller/machine_safety_test.go
+++ b/pkg/controller/machine_safety_test.go
@@ -20,9 +20,11 @@ import (
 
 	"github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
 	machinev1 "github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
+	"github.com/gardener/machine-controller-manager/pkg/driver"
 	. "github.com/onsi/ginkgo"
 	. "github.com/onsi/ginkgo/extensions/table"
 	. "github.com/onsi/gomega"
+	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/runtime"
@@ -781,3 +783,109 @@ var _ = Describe("#machine_safety", func() {
 			false, false, fiveMinutesDuration, true, true),
 	)
 })
+
+var _ = Describe("machineCrashloopBackoff", func() {
+	objMeta := &metav1.ObjectMeta{
+		GenerateName: "class",
+		Namespace:    testNamespace,
+	}
+
+	classKind := "MachineClass"
+	secretData := map[string][]byte{
+		"userData":            []byte("dummy-data"),
+		"azureClientId":       []byte("dummy-client-id"),
+		"azureClientSecret":   []byte("dummy-client-secret"),
+		"azureSubscriptionId": []byte("dummy-subcription-id"),
+		"azureTenantId":       []byte("dummy-tenant-id"),
+	}
+
+	Describe("machineCrashloopBackoff", func() {
+
+		It("Should delete the machine (old code)", func() {
+			stop := make(chan struct{})
+			defer close(stop)
+
+			// Create test secret and add it to controlCoreObject list
+			testSecret := &v1.Secret{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "test-secret",
+					Namespace: testNamespace,
+				},
+				Data: secretData,
+			}
+
+			// Create a test secretReference because the method checkMachineClass needs it
+			testSecretReference := &v1.SecretReference{
+				Name:      "test-secret",
+				Namespace: testNamespace,
+			}
+
+			testMachineClass := &machinev1.MachineClass{
+				ObjectMeta: *newObjectMeta(objMeta, 0),
+				SecretRef:  testSecretReference,
+			}
+			controlCoreObjects := []runtime.Object{}
+			controlCoreObjects = append(controlCoreObjects, testSecret)
+
+			// Create test machine object in CrashloopBackoff state
+			testMachineObject1 := &machinev1.Machine{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "testmachine_1",
+					Namespace: testNamespace,
+				},
+				Status: machinev1.MachineStatus{
+					CurrentStatus: machinev1.CurrentStatus{
+						Phase: v1alpha1.MachineCrashLoopBackOff,
+					},
+				},
+			}
+
+			// Create another test machine object in Running state
+			testMachineObject2 := &machinev1.Machine{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "testmachine_2",
+					Namespace: testNamespace,
+				},
+				Status: machinev1.MachineStatus{
+					CurrentStatus: machinev1.CurrentStatus{
+						Phase: v1alpha1.MachineRunning,
+					},
+				},
+			}
+
+			controlMachineObjects := []runtime.Object{}
+			controlMachineObjects = append(controlMachineObjects, testMachineObject1)
+			controlMachineObjects = append(controlMachineObjects, testMachineObject2)
+
+			c, trackers := createController(stop, testNamespace, controlMachineObjects, controlCoreObjects, nil)
+			defer trackers.Stop()
+
+			fakeDriver := driver.NewDriver(
+				"",
+				secretData,
+				classKind,
+				testMachineClass,
+				"",
+			)
+
+			// use type assertion to let Golang know that the
+			// returned fakeDriver is of type *driver.FakeDriver
+			fd := fakeDriver.(*driver.FakeDriver)
+
+			_ = fd.Add("testmachine-ip1", "testmachine_1")
+			_ = fd.Add("testmachine-ip2", "testmachine_2")
+
+			waitForCacheSync(stop, c)
+
+			// call checkMachineClass to delete the orphan VMs
+			c.checkMachineClass(testMachineClass, testSecretReference, testSecretReference, objMeta.Name, classKind)
+
+			listOfVMs, _ := fakeDriver.GetVMs("")
+
+			// after this, the testmachine in crashloopbackoff phase should remain and the other one should
+			// be deleted because it is an orphan VM
+			Expect(listOfVMs["testmachine-ip1"]).To(Equal("testmachine_1"))
+			Expect(listOfVMs["testmachine-ip2"]).To(Equal(""))
+		})
+	})
+})
diff --git a/pkg/controller/machine_test.go b/pkg/controller/machine_test.go
index 9262a8dc4..74367238d 100644
--- a/pkg/controller/machine_test.go
+++ b/pkg/controller/machine_test.go
@@ -684,12 +684,15 @@ var _ = Describe("machine", func() {
 					func() (string, string, error) {
 						return action.fakeProviderID, action.fakeNodeName, action.fakeError
 					},
+					func(string, string) error {
+						return nil
+					},
 					func(string) error {
 						return action.fakeError
 					}, nil,
 					func() (driver.VMs, error) {
 						return map[string]string{}, nil
-					},
+					}, nil, nil, nil,
 				))
 
 				if data.expect.err {
@@ -1052,6 +1055,9 @@ var _ = Describe("machine", func() {
 						}
 						return action.fakeProviderID, action.fakeNodeName, action.fakeError
 					},
+					func(string, string) error {
+						return nil
+					},
 					func(string) error {
 						return nil
 					},
@@ -1059,6 +1065,7 @@ var _ = Describe("machine", func() {
 						return action.fakeProviderID, action.fakeError
 					},
 					fakeDriverGetVMsTemp,
+					nil, nil, nil,
 				)
 
 				// Create a machine that is to be deleted later
diff --git a/pkg/util/provider/machinecontroller/machine_safety_test.go b/pkg/util/provider/machinecontroller/machine_safety_test.go
index 29a07ef6a..1033ee4f7 100644
--- a/pkg/util/provider/machinecontroller/machine_safety_test.go
+++ b/pkg/util/provider/machinecontroller/machine_safety_test.go
@@ -16,13 +16,16 @@ limitations under the License.
 package controller
 
 import (
+	"context"
 	"time"
 
 	"github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
 	machinev1 "github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
+	"github.com/gardener/machine-controller-manager/pkg/util/provider/driver"
 	. "github.com/onsi/ginkgo"
 	. "github.com/onsi/ginkgo/extensions/table"
 	. "github.com/onsi/gomega"
+	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 )
@@ -136,3 +139,107 @@ var _ = Describe("#machine_safety", func() {
 			false, false, fiveMinutesDuration, true, true),
 	)
 })
+
+var _ = Describe("machineCrashloopBackoff", func() {
+	objMeta := &metav1.ObjectMeta{
+		GenerateName: "class",
+		Namespace:    testNamespace,
+	}
+
+	// classKind := "MachineClass"
+	secretData := map[string][]byte{
+		"userData":            []byte("dummy-data"),
+		"azureClientId":       []byte("dummy-client-id"),
+		"azureClientSecret":   []byte("dummy-client-secret"),
+		"azureSubscriptionId": []byte("dummy-subcription-id"),
+		"azureTenantId":       []byte("dummy-tenant-id"),
+	}
+
+	Describe("machineCrashloopBackoff", func() {
+
+		It("Should delete the machine (old code)", func() {
+			stop := make(chan struct{})
+			defer close(stop)
+
+			// Create test secret and add it to controlCoreObject list
+			testSecret := &v1.Secret{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "test-secret",
+					Namespace: testNamespace,
+				},
+				Data: secretData,
+			}
+
+			// Create a test secretReference because the method checkMachineClass needs it
+			testSecretReference := &v1.SecretReference{
+				Name:      "test-secret",
+				Namespace: testNamespace,
+			}
+
+			testMachineClass := &machinev1.MachineClass{
+				ObjectMeta: *newObjectMeta(objMeta, 0),
+				SecretRef:  testSecretReference,
+			}
+
+			controlCoreObjects := []runtime.Object{}
+			controlCoreObjects = append(controlCoreObjects, testSecret)
+
+			// Create test machine object in CrashloopBackoff state
+			testMachineObject1 := &machinev1.Machine{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "testmachine_1",
+					Namespace: testNamespace,
+				},
+				Status: machinev1.MachineStatus{
+					CurrentStatus: machinev1.CurrentStatus{
+						Phase: v1alpha1.MachineCrashLoopBackOff,
+					},
+				},
+			}
+
+			// Create another test machine object in Running state
+			testMachineObject2 := &machinev1.Machine{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "testmachine_2",
+					Namespace: testNamespace,
+				},
+				Status: machinev1.MachineStatus{
+					CurrentStatus: machinev1.CurrentStatus{
+						Phase: v1alpha1.MachineRunning,
+					},
+				},
+			}
+
+			controlMachineObjects := []runtime.Object{}
+			controlMachineObjects = append(controlMachineObjects, testMachineObject1)
+			controlMachineObjects = append(controlMachineObjects, testMachineObject2)
+
+			fakeDriver := driver.NewFakeDriver(false, "", "", "", nil, nil)
+
+			c, trackers := createController(stop, testNamespace, controlMachineObjects, controlCoreObjects, nil, fakeDriver)
+			defer trackers.Stop()
+
+			fd := fakeDriver.(*driver.FakeDriver)
+
+			listMachinesRequest := &driver.ListMachinesRequest{
+				MachineClass: testMachineClass,
+				Secret:       testSecret,
+			}
+
+			_ = fd.AddMachine("testmachine-ip1", "testmachine_1")
+			_ = fd.AddMachine("testmachine-ip2", "testmachine_2")
+			waitForCacheSync(stop, c)
+
+			// call checkMachineClass to delete the orphan VMs
+			_, _ = c.checkMachineClass(testMachineClass)
+
+			// after this, the testmachine in crashloopbackoff phase
+			// should remain and the other one should
+			// be deleted because it is an orphan VM
+			listMachinesResponse, _ := fd.ListMachines(context.Background(), listMachinesRequest)
+
+			Expect(listMachinesResponse.MachineList["testmachine-ip1"]).To(Equal("testmachine_1"))
+			Expect(listMachinesResponse.MachineList["testmachine-ip2"]).To(Equal(""))
+		})
+	})
+})
diff --git a/pkg/util/provider/machinecontroller/machine_test.go b/pkg/util/provider/machinecontroller/machine_test.go
index b7709c839..58392e07a 100644
--- a/pkg/util/provider/machinecontroller/machine_test.go
+++ b/pkg/util/provider/machinecontroller/machine_test.go
@@ -392,7 +392,14 @@ var _ = Describe("machine", func() {
 					coreObjects = append(coreObjects, o)
 				}
 
-				fakedriver := driver.NewFakeDriver(data.action.fakeDriver)
+				fakedriver := driver.NewFakeDriver(
+					data.action.fakeDriver.VMExists,
+					data.action.fakeDriver.ProviderID,
+					data.action.fakeDriver.NodeName,
+					data.action.fakeDriver.LastKnownState,
+					data.action.fakeDriver.Err,
+					nil,
+				)
 
 				controller, trackers := createController(stop, objMeta.Namespace, machineObjects, nil, coreObjects, fakedriver)
 				defer trackers.Stop()
@@ -915,7 +922,12 @@ var _ = Describe("machine", func() {
 				}
 
 				fakeDriver := driver.NewFakeDriver(
-					data.action.fakeDriver,
+					data.action.fakeDriver.VMExists,
+					data.action.fakeDriver.ProviderID,
+					data.action.fakeDriver.NodeName,
+					data.action.fakeDriver.LastKnownState,
+					data.action.fakeDriver.Err,
+					nil,
 				)
 
 				controller, trackers := createController(stop, objMeta.Namespace, machineObjects, nil, coreObjects, fakeDriver)
diff --git a/pkg/util/provider/machinecontroller/machineclass_test.go b/pkg/util/provider/machinecontroller/machineclass_test.go
index 5535967cf..6ba6b4c02 100644
--- a/pkg/util/provider/machinecontroller/machineclass_test.go
+++ b/pkg/util/provider/machinecontroller/machineclass_test.go
@@ -75,7 +75,12 @@ var _ = Describe("machineclass", func() {
 				}
 
 				fakeDriver := driver.NewFakeDriver(
-					data.action.fakeDriver,
+					data.action.fakeDriver.VMExists,
+					data.action.fakeDriver.ProviderID,
+					data.action.fakeDriver.NodeName,
+					data.action.fakeDriver.LastKnownState,
+					data.action.fakeDriver.Err,
+					nil,
 				)
 
 				controller, trackers := createController(stop, TestNamespace, machineObjects, nil, nil, fakeDriver)
diff --git a/pkg/util/provider/machinecontroller/migrate_machineclass_test.go b/pkg/util/provider/machinecontroller/migrate_machineclass_test.go
index 6b3d20b07..544a69d4d 100644
--- a/pkg/util/provider/machinecontroller/migrate_machineclass_test.go
+++ b/pkg/util/provider/machinecontroller/migrate_machineclass_test.go
@@ -157,7 +157,14 @@ var _ = Describe("machine", func() {
 					machineObjects = append(machineObjects, o)
 				}
 
-				fakedriver := driver.NewFakeDriver(data.action.fakeDriver)
+				fakedriver := driver.NewFakeDriver(
+					data.action.fakeDriver.VMExists,
+					data.action.fakeDriver.ProviderID,
+					data.action.fakeDriver.NodeName,
+					data.action.fakeDriver.LastKnownState,
+					data.action.fakeDriver.Err,
+					nil,
+				)
 
 				controller, trackers := createController(stop, TestNamespace, machineObjects, nil, nil, fakedriver)
 				defer trackers.Stop()