Repository: kubernetes-sigs/cluster-api-provider-kubevirt (go)

Issue:
cluster teardown hangs
/kind bug

A kubevirt provider cluster should be able to be torn down by deleting the top level Cluster object, however that doesn't work today.

The issue is that the KubeVirtMachine objects will never have their finalizer removed during the cascading deletion if the top level Cluster object get's deleted before the KubeVirtMachines can be processed.

The result is that the KubeVirtMachine objects block indefinitely trying to get the non-existent Cluster object as a precondition to processing the deletion and removing the finalizer. This blocks the cluster from being torn down entirely. 

Pull request description:
Cluster deletion race
Fixes #65 

# Overview
A KubevirtMachine object should be able to successfully terminate and be removed regardless if the cooresponding Cluster or KubevirtCluster object exists.

Today, we have no strong guarantee that a corresponding KubevirtCluster will still exist when a KubevirtMachine is being torn down. If the kv cluster object disappears before the kv machine, the machine controller will block indefinitely waiting for the kv cluster object to be accessible before processing the kv machine's deletion.

# Steps to Reproduce

Delete a KubevirtCluster's namespace and the kubevirt machines will block the namespace from being terminated indefinitely.

# Fix Explanation

The KubevirtMachine objects depended on the KubevirtCluster in order to retrieve a reference to a secret that represents the kubeconfig used to create/delete VMs.  If the KubevirtCluster disappears before the KubevirtMachine, then the kv machine can not process the VM deletion accurately because it doesn't know the secret ref to use.

To fix this, the KubevirtMachine object now stores the secret ref itself on its spec in the `KubevirtMachine.Spec.InfraClusterSecretRef ` field similar to the KubevirtCluster. This value will default to the cluster's secret ref when nil, which preserves existing behavior.

By storing the secret ref on the kv machine itself, the machine can be torn down independently of whether or not the kv cluster still exists.

# Alternate solutions

An alternate solution to this that was given consideration was to put a finalizer on the KubevirtCluster object and not allow it to be deleted until all corresponding KubevirtMachines are gone. The problem with this is KubevirtCluster's which are externally managed outside of the capk controllers #79. When a KubevirtCluster is externally managed, the capk controller won't reconcile it, which means we don't maintain control over a finalizer or it's usage.

```release-note
Fixes race condition between kubevirtcluster and kubevirtmachine during tear down
```


Patch:
diff --git a/api/v1alpha1/kubevirtmachine_types.go b/api/v1alpha1/kubevirtmachine_types.go
index 13e3cdc..dcd9e2b 100644
--- a/api/v1alpha1/kubevirtmachine_types.go
+++ b/api/v1alpha1/kubevirtmachine_types.go
@@ -17,6 +17,7 @@ limitations under the License.
 package v1alpha1
 
 import (
+	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	kubevirtv1 "kubevirt.io/api/core/v1"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
@@ -44,6 +45,11 @@ type KubevirtMachineSpec struct {
 	// ProviderID TBD what to use for Kubevirt
 	// +optional
 	ProviderID *string `json:"providerID,omitempty"`
+
+	// InfraClusterSecretRef is a reference to a secret with a kubeconfig for external cluster used for infra.
+	// When nil, this defaults to the value present in the KubevirtCluster object's spec associated with this machine.
+	// +optional
+	InfraClusterSecretRef *corev1.ObjectReference `json:"infraClusterSecretRef,omitempty"`
 }
 
 // KubevirtMachineStatus defines the observed state of KubevirtMachine.
diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go
index 05832c7..9812c0b 100644
--- a/api/v1alpha1/zz_generated.deepcopy.go
+++ b/api/v1alpha1/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
@@ -238,6 +237,11 @@ func (in *KubevirtMachineSpec) DeepCopyInto(out *KubevirtMachineSpec) {
 		*out = new(string)
 		**out = **in
 	}
+	if in.InfraClusterSecretRef != nil {
+		in, out := &in.InfraClusterSecretRef, &out.InfraClusterSecretRef
+		*out = new(v1.ObjectReference)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KubevirtMachineSpec.
diff --git a/config/crd/bases/infrastructure.cluster.x-k8s.io_kubevirtmachines.yaml b/config/crd/bases/infrastructure.cluster.x-k8s.io_kubevirtmachines.yaml
index 756e139..44e7347 100644
--- a/config/crd/bases/infrastructure.cluster.x-k8s.io_kubevirtmachines.yaml
+++ b/config/crd/bases/infrastructure.cluster.x-k8s.io_kubevirtmachines.yaml
@@ -38,6 +38,45 @@ spec:
           spec:
             description: KubevirtMachineSpec defines the desired state of KubevirtMachine.
             properties:
+              infraClusterSecretRef:
+                description: InfraClusterSecretRef is a reference to a secret with
+                  a kubeconfig for external cluster used for infra. When nil, this
+                  defaults to the value present in the KubevirtCluster object's spec
+                  associated with this machine.
+                properties:
+                  apiVersion:
+                    description: API version of the referent.
+                    type: string
+                  fieldPath:
+                    description: 'If referring to a piece of an object instead of
+                      an entire object, this string should contain a valid JSON/Go
+                      field access statement, such as desiredState.manifest.containers[2].
+                      For example, if the object reference is to a container within
+                      a pod, this would take on a value like: "spec.containers{name}"
+                      (where "name" refers to the name of the container that triggered
+                      the event) or if no container name is specified "spec.containers[2]"
+                      (container with index 2 in this pod). This syntax is chosen
+                      only to have some well-defined way of referencing a part of
+                      an object. TODO: this design is not final and this field is
+                      subject to change in the future.'
+                    type: string
+                  kind:
+                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                    type: string
+                  name:
+                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                    type: string
+                  namespace:
+                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                    type: string
+                  resourceVersion:
+                    description: 'Specific resourceVersion to which this reference
+                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                    type: string
+                  uid:
+                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                    type: string
+                type: object
               providerID:
                 description: ProviderID TBD what to use for Kubevirt
                 type: string
diff --git a/config/crd/bases/infrastructure.cluster.x-k8s.io_kubevirtmachinetemplates.yaml b/config/crd/bases/infrastructure.cluster.x-k8s.io_kubevirtmachinetemplates.yaml
index eac15f5..19008ec 100644
--- a/config/crd/bases/infrastructure.cluster.x-k8s.io_kubevirtmachinetemplates.yaml
+++ b/config/crd/bases/infrastructure.cluster.x-k8s.io_kubevirtmachinetemplates.yaml
@@ -48,6 +48,46 @@ spec:
                     description: Spec is the specification of the desired behavior
                       of the machine.
                     properties:
+                      infraClusterSecretRef:
+                        description: InfraClusterSecretRef is a reference to a secret
+                          with a kubeconfig for external cluster used for infra. When
+                          nil, this defaults to the value present in the KubevirtCluster
+                          object's spec associated with this machine.
+                        properties:
+                          apiVersion:
+                            description: API version of the referent.
+                            type: string
+                          fieldPath:
+                            description: 'If referring to a piece of an object instead
+                              of an entire object, this string should contain a valid
+                              JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                              For example, if the object reference is to a container
+                              within a pod, this would take on a value like: "spec.containers{name}"
+                              (where "name" refers to the name of the container that
+                              triggered the event) or if no container name is specified
+                              "spec.containers[2]" (container with index 2 in this
+                              pod). This syntax is chosen only to have some well-defined
+                              way of referencing a part of an object. TODO: this design
+                              is not final and this field is subject to change in
+                              the future.'
+                            type: string
+                          kind:
+                            description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                            type: string
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                            type: string
+                          namespace:
+                            description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                            type: string
+                          resourceVersion:
+                            description: 'Specific resourceVersion to which this reference
+                              is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                            type: string
+                          uid:
+                            description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                            type: string
+                        type: object
                       providerID:
                         description: ProviderID TBD what to use for Kubevirt
                         type: string
diff --git a/controllers/kubevirtcluster_controller.go b/controllers/kubevirtcluster_controller.go
index b0f2b3d..77e0499 100644
--- a/controllers/kubevirtcluster_controller.go
+++ b/controllers/kubevirtcluster_controller.go
@@ -87,7 +87,7 @@ func (r *KubevirtClusterReconciler) Reconcile(goctx gocontext.Context, req ctrl.
 		Logger:          ctrl.LoggerFrom(goctx).WithName(req.Namespace).WithName(req.Name),
 	}
 
-	infraClusterClient, infraClusterNamespace, err := r.InfraCluster.GenerateInfraClusterClient(clusterContext)
+	infraClusterClient, infraClusterNamespace, err := r.InfraCluster.GenerateInfraClusterClient(kubevirtCluster.Spec.InfraClusterSecretRef, kubevirtCluster.Namespace, goctx)
 	if err != nil {
 		return ctrl.Result{RequeueAfter: 10 * time.Second}, errors.Wrap(err, "failed to generate infra cluster client")
 	}
diff --git a/controllers/kubevirtmachine_controller.go b/controllers/kubevirtmachine_controller.go
index bc68a74..15ee772 100644
--- a/controllers/kubevirtmachine_controller.go
+++ b/controllers/kubevirtmachine_controller.go
@@ -91,6 +91,21 @@ func (r *KubevirtMachineReconciler) Reconcile(goctx gocontext.Context, req ctrl.
 
 	log = log.WithValues("machine", machine.Name)
 
+	// Handle deleted machines
+	if !kubevirtMachine.ObjectMeta.DeletionTimestamp.IsZero() {
+		// Create the machine context for this request.
+		// Deletion shouldn't require the presence of a
+		// cluster or kubevirtcluster object as those objects
+		// may have already been removed.
+		machineContext := &context.MachineContext{
+			Context:         goctx,
+			Machine:         machine,
+			KubevirtMachine: kubevirtMachine,
+			Logger:          ctrl.LoggerFrom(goctx).WithName(req.Namespace).WithName(req.Name),
+		}
+		return r.reconcileDelete(machineContext)
+	}
+
 	// Fetch the Cluster.
 	cluster, err := util.GetClusterFromMetadata(goctx, r.Client, machine.ObjectMeta)
 	if err != nil {
@@ -156,11 +171,6 @@ func (r *KubevirtMachineReconciler) Reconcile(goctx gocontext.Context, req ctrl.
 		return ctrl.Result{}, nil
 	}
 
-	// Handle deleted machines
-	if !kubevirtMachine.ObjectMeta.DeletionTimestamp.IsZero() {
-		return r.reconcileDelete(machineContext)
-	}
-
 	// Handle non-deleted machines
 	res, err := r.reconcileNormal(machineContext)
 
@@ -203,7 +213,13 @@ func (r *KubevirtMachineReconciler) reconcileNormal(ctx *context.MachineContext)
 		}
 	}
 
-	infraClusterClient, infraClusterNamespace, err := r.InfraCluster.GenerateInfraClusterClient(ctx.ClusterContext())
+	// Default the infra cluster secret ref when the
+	// machine does not have one set.
+	if ctx.KubevirtMachine.Spec.InfraClusterSecretRef == nil {
+		ctx.KubevirtMachine.Spec.InfraClusterSecretRef = ctx.KubevirtCluster.Spec.InfraClusterSecretRef
+	}
+
+	infraClusterClient, infraClusterNamespace, err := r.InfraCluster.GenerateInfraClusterClient(ctx.KubevirtMachine.Spec.InfraClusterSecretRef, ctx.KubevirtMachine.Namespace, ctx.Context)
 	if err != nil {
 		return ctrl.Result{RequeueAfter: 10 * time.Second}, errors.Wrap(err, "failed to generate infra cluster client")
 	}
@@ -362,7 +378,13 @@ func (r *KubevirtMachineReconciler) updateNodeProviderID(ctx *context.MachineCon
 }
 
 func (r *KubevirtMachineReconciler) reconcileDelete(ctx *context.MachineContext) (ctrl.Result, error) {
-	infraClusterClient, vmNamespace, err := r.InfraCluster.GenerateInfraClusterClient(ctx.ClusterContext())
+
+	patchHelper, err := patch.NewHelper(ctx.KubevirtMachine, r.Client)
+	if err != nil {
+		return ctrl.Result{}, err
+	}
+
+	infraClusterClient, infraClusterNamespace, err := r.InfraCluster.GenerateInfraClusterClient(ctx.KubevirtMachine.Spec.InfraClusterSecretRef, ctx.KubevirtMachine.Namespace, ctx.Context)
 	if err != nil {
 		return ctrl.Result{RequeueAfter: 10 * time.Second}, errors.Wrap(err, "failed to generate infra cluster client")
 	}
@@ -371,6 +393,16 @@ func (r *KubevirtMachineReconciler) reconcileDelete(ctx *context.MachineContext)
 		return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
 	}
 
+	// If there is not a namespace explicitly set on the vm template, then
+	// use the infra namespace as a default. For internal clusters, the infraNamespace
+	// will be the same as the KubeVirtCluster object, for external clusters the
+	// infraNamespace will attempt to be detected from the infraClusterSecretRef's
+	// kubeconfig
+	vmNamespace := ctx.KubevirtMachine.Spec.VirtualMachineTemplate.ObjectMeta.Namespace
+	if vmNamespace == "" {
+		vmNamespace = infraClusterNamespace
+	}
+
 	ctx.Logger.Info("Deleting VM bootstrap secret...")
 	if err := r.deleteKubevirtBootstrapSecret(ctx, infraClusterClient, vmNamespace); err != nil {
 		return ctrl.Result{RequeueAfter: 10 * time.Second}, errors.Wrap(err, "failed to delete bootstrap secret")
@@ -387,20 +419,16 @@ func (r *KubevirtMachineReconciler) reconcileDelete(ctx *context.MachineContext)
 		}
 	}
 
+	// Machine is deleted so remove the finalizer.
+	controllerutil.RemoveFinalizer(ctx.KubevirtMachine, infrav1.MachineFinalizer)
+
 	// Set the VMProvisionedCondition reporting delete is started, and issue a patch in order to make
 	// this visible to the users.
-	patchHelper, err := patch.NewHelper(ctx.KubevirtMachine, r.Client)
-	if err != nil {
-		return ctrl.Result{}, err
-	}
 	conditions.MarkFalse(ctx.KubevirtMachine, infrav1.VMProvisionedCondition, clusterv1.DeletingReason, clusterv1.ConditionSeverityInfo, "")
 	if err := ctx.PatchKubevirtMachine(patchHelper); err != nil {
 		return ctrl.Result{}, errors.Wrap(err, "failed to patch KubevirtMachine")
 	}
 
-	// Machine is deleted so remove the finalizer.
-	controllerutil.RemoveFinalizer(ctx.KubevirtMachine, infrav1.MachineFinalizer)
-
 	return ctrl.Result{}, nil
 }
 
diff --git a/go.mod b/go.mod
index 4be4737..d86efa2 100644
--- a/go.mod
+++ b/go.mod
@@ -55,12 +55,15 @@ require (
 	go.uber.org/atomic v1.7.0 // indirect
 	go.uber.org/multierr v1.6.0 // indirect
 	go.uber.org/zap v1.19.1 // indirect
+	golang.org/x/mod v0.4.2 // indirect
 	golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2 // indirect
 	golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f // indirect
 	golang.org/x/sys v0.0.0-20211029165221-6e7872819dc8 // indirect
 	golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b // indirect
 	golang.org/x/text v0.3.6 // indirect
 	golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac // indirect
+	golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
 	gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/protobuf v1.27.1 // indirect
diff --git a/go.sum b/go.sum
index 545004f..ee63b51 100644
--- a/go.sum
+++ b/go.sum
@@ -728,6 +728,7 @@ golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.4.2 h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo=
 golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -986,6 +987,7 @@ golang.org/x/tools v0.1.2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
 golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
 golang.org/x/tools v0.1.4/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
 golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
+golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff h1:VX/uD7MK0AHXGiScH3fsieUQUcpmRERPDYtqZdJnA+Q=
 golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff/go.mod h1:YD9qOF0M9xpSpdWTBbzEl5e/RnCefISl8E5Noe10jFM=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
diff --git a/pkg/infracluster/infracluster.go b/pkg/infracluster/infracluster.go
index 8a35a9e..2dfe71b 100644
--- a/pkg/infracluster/infracluster.go
+++ b/pkg/infracluster/infracluster.go
@@ -1,19 +1,18 @@
 package infracluster
 
 import (
+	gocontext "context"
 	"strings"
 
 	"github.com/pkg/errors"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/client-go/tools/clientcmd"
 	"sigs.k8s.io/controller-runtime/pkg/client"
-
-	"sigs.k8s.io/cluster-api-provider-kubevirt/pkg/context"
 )
 
 //go:generate mockgen -source=./infracluster.go -destination=./mock/infracluster_generated.go -package=mock
 type InfraCluster interface {
-	GenerateInfraClusterClient(ctx *context.ClusterContext) (client.Client, string, error)
+	GenerateInfraClusterClient(infraClusterSecretRef *corev1.ObjectReference, ownerNamespace string, context gocontext.Context) (client.Client, string, error)
 }
 
 // New creates new InfraCluster instance
@@ -28,16 +27,14 @@ type infraCluster struct {
 }
 
 // GenerateInfraClusterClient creates a client for infra cluster.
-func (w *infraCluster) GenerateInfraClusterClient(ctx *context.ClusterContext) (client.Client, string, error) {
-	infraClusterSecretRef := ctx.KubevirtCluster.Spec.InfraClusterSecretRef
-
+func (w *infraCluster) GenerateInfraClusterClient(infraClusterSecretRef *corev1.ObjectReference, ownerNamespace string, context gocontext.Context) (client.Client, string, error) {
 	if infraClusterSecretRef == nil {
-		return w.Client, ctx.Cluster.Namespace, nil
+		return w.Client, ownerNamespace, nil
 	}
 
 	infraKubeconfigSecret := &corev1.Secret{}
 	infraKubeconfigSecretKey := client.ObjectKey{Namespace: infraClusterSecretRef.Namespace, Name: infraClusterSecretRef.Name}
-	if err := w.Client.Get(ctx.Context, infraKubeconfigSecretKey, infraKubeconfigSecret); err != nil {
+	if err := w.Client.Get(context, infraKubeconfigSecretKey, infraKubeconfigSecret); err != nil {
 		return nil, "", errors.Wrapf(err, "failed to fetch infra kubeconfig secret %s/%s", infraClusterSecretRef.Namespace, infraClusterSecretRef.Name)
 	}
 
diff --git a/pkg/infracluster/mock/infracluster_generated.go b/pkg/infracluster/mock/infracluster_generated.go
index c6d29b6..8df2e1b 100644
--- a/pkg/infracluster/mock/infracluster_generated.go
+++ b/pkg/infracluster/mock/infracluster_generated.go
@@ -5,12 +5,12 @@
 package mock
 
 import (
-	"reflect"
+	context "context"
+	reflect "reflect"
 
-	"github.com/golang/mock/gomock"
-	"sigs.k8s.io/controller-runtime/pkg/client"
-
-	"sigs.k8s.io/cluster-api-provider-kubevirt/pkg/context"
+	gomock "github.com/golang/mock/gomock"
+	v1 "k8s.io/api/core/v1"
+	client "sigs.k8s.io/controller-runtime/pkg/client"
 )
 
 // MockInfraCluster is a mock of InfraCluster interface.
@@ -37,9 +37,9 @@ func (m *MockInfraCluster) EXPECT() *MockInfraClusterMockRecorder {
 }
 
 // GenerateInfraClusterClient mocks base method.
-func (m *MockInfraCluster) GenerateInfraClusterClient(ctx *context.ClusterContext) (client.Client, string, error) {
+func (m *MockInfraCluster) GenerateInfraClusterClient(infraClusterSecretRef *v1.ObjectReference, ownerNamespace string, context context.Context) (client.Client, string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GenerateInfraClusterClient", ctx)
+	ret := m.ctrl.Call(m, "GenerateInfraClusterClient", infraClusterSecretRef, ownerNamespace, context)
 	ret0, _ := ret[0].(client.Client)
 	ret1, _ := ret[1].(string)
 	ret2, _ := ret[2].(error)
@@ -47,7 +47,7 @@ func (m *MockInfraCluster) GenerateInfraClusterClient(ctx *context.ClusterContex
 }
 
 // GenerateInfraClusterClient indicates an expected call of GenerateInfraClusterClient.
-func (mr *MockInfraClusterMockRecorder) GenerateInfraClusterClient(ctx interface{}) *gomock.Call {
+func (mr *MockInfraClusterMockRecorder) GenerateInfraClusterClient(infraClusterSecretRef, ownerNamespace, context interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GenerateInfraClusterClient", reflect.TypeOf((*MockInfraCluster)(nil).GenerateInfraClusterClient), ctx)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GenerateInfraClusterClient", reflect.TypeOf((*MockInfraCluster)(nil).GenerateInfraClusterClient), infraClusterSecretRef, ownerNamespace, context)
 }
diff --git a/pkg/workloadcluster/mock/workloadcluster_generated.go b/pkg/workloadcluster/mock/workloadcluster_generated.go
index d70c4b2..53b63e9 100644
--- a/pkg/workloadcluster/mock/workloadcluster_generated.go
+++ b/pkg/workloadcluster/mock/workloadcluster_generated.go
@@ -13,30 +13,30 @@ import (
 	context "sigs.k8s.io/cluster-api-provider-kubevirt/pkg/context"
 )
 
-// MockWorkloadCluster is a mock of WorkloadCluster interface
+// MockWorkloadCluster is a mock of WorkloadCluster interface.
 type MockWorkloadCluster struct {
 	ctrl     *gomock.Controller
 	recorder *MockWorkloadClusterMockRecorder
 }
 
-// MockWorkloadClusterMockRecorder is the mock recorder for MockWorkloadCluster
+// MockWorkloadClusterMockRecorder is the mock recorder for MockWorkloadCluster.
 type MockWorkloadClusterMockRecorder struct {
 	mock *MockWorkloadCluster
 }
 
-// NewMockWorkloadCluster creates a new mock instance
+// NewMockWorkloadCluster creates a new mock instance.
 func NewMockWorkloadCluster(ctrl *gomock.Controller) *MockWorkloadCluster {
 	mock := &MockWorkloadCluster{ctrl: ctrl}
 	mock.recorder = &MockWorkloadClusterMockRecorder{mock}
 	return mock
 }
 
-// EXPECT returns an object that allows the caller to indicate expected use
+// EXPECT returns an object that allows the caller to indicate expected use.
 func (m *MockWorkloadCluster) EXPECT() *MockWorkloadClusterMockRecorder {
 	return m.recorder
 }
 
-// GenerateWorkloadClusterClient mocks base method
+// GenerateWorkloadClusterClient mocks base method.
 func (m *MockWorkloadCluster) GenerateWorkloadClusterClient(ctx *context.MachineContext) (client.Client, error) {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "GenerateWorkloadClusterClient", ctx)
@@ -45,7 +45,7 @@ func (m *MockWorkloadCluster) GenerateWorkloadClusterClient(ctx *context.Machine
 	return ret0, ret1
 }
 
-// GenerateWorkloadClusterClient indicates an expected call of GenerateWorkloadClusterClient
+// GenerateWorkloadClusterClient indicates an expected call of GenerateWorkloadClusterClient.
 func (mr *MockWorkloadClusterMockRecorder) GenerateWorkloadClusterClient(ctx interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GenerateWorkloadClusterClient", reflect.TypeOf((*MockWorkloadCluster)(nil).GenerateWorkloadClusterClient), ctx)


Test patch:
diff --git a/controllers/kubevirtcluster_controller_test.go b/controllers/kubevirtcluster_controller_test.go
index d9c53f2..99565ad 100644
--- a/controllers/kubevirtcluster_controller_test.go
+++ b/controllers/kubevirtcluster_controller_test.go
@@ -67,7 +67,7 @@ var _ = Describe("Reconcile", func() {
 				kubevirtCluster,
 			}
 			setupClient(objects)
-			infraClusterMock.EXPECT().GenerateInfraClusterClient(gomock.Any()).Return(fakeClient, kubevirtCluster.Namespace, nil)
+			infraClusterMock.EXPECT().GenerateInfraClusterClient(gomock.Any(), gomock.Any(), gomock.Any()).Return(fakeClient, kubevirtCluster.Namespace, nil)
 
 			result, err := kubevirtClusterReconciler.Reconcile(fakeContext, Request{
 				NamespacedName: client.ObjectKey{
@@ -122,7 +122,7 @@ var _ = Describe("Reconcile", func() {
 				kubevirtCluster,
 			}
 			setupClient(objects)
-			infraClusterMock.EXPECT().GenerateInfraClusterClient(gomock.Any()).Return(fakeClient, kubevirtCluster.Namespace, nil)
+			infraClusterMock.EXPECT().GenerateInfraClusterClient(gomock.Any(), gomock.Any(), gomock.Any()).Return(fakeClient, kubevirtCluster.Namespace, nil)
 
 			_, err := kubevirtClusterReconciler.Reconcile(fakeContext, Request{
 				NamespacedName: client.ObjectKey{
@@ -153,7 +153,7 @@ var _ = Describe("Reconcile", func() {
 				kubevirtCluster,
 			}
 			setupClient(objects)
-			infraClusterMock.EXPECT().GenerateInfraClusterClient(gomock.Any()).Return(fakeClient, kubevirtCluster.Namespace, nil)
+			infraClusterMock.EXPECT().GenerateInfraClusterClient(gomock.Any(), gomock.Any(), gomock.Any()).Return(fakeClient, kubevirtCluster.Namespace, nil)
 
 			_, err := kubevirtClusterReconciler.Reconcile(fakeContext, Request{
 				NamespacedName: client.ObjectKey{
diff --git a/controllers/kubevirtmachine_controller_test.go b/controllers/kubevirtmachine_controller_test.go
index d36c895..be6d04b 100644
--- a/controllers/kubevirtmachine_controller_test.go
+++ b/controllers/kubevirtmachine_controller_test.go
@@ -39,6 +39,7 @@ import (
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 
 	machinemocks "sigs.k8s.io/cluster-api-provider-kubevirt/pkg/kubevirt/mock"
 
@@ -264,8 +265,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 		setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-		clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-		infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 		out, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 
@@ -306,8 +306,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 		machineMock.EXPECT().GenerateProviderID().Return("abc", nil).Times(1)
 		machineFactoryMock.EXPECT().NewMachine(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(machineMock, nil).Times(1)
 
-		clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-		infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil).Times(3)
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil).Times(3)
 
 		out, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 
@@ -323,7 +322,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 		//Check bootstrapData secret is deleted
 		machineBootstrapSecretReferenceName := machineContext.Machine.Spec.Bootstrap.DataSecretName
 		machineBootstrapSecretReferenceKey := client.ObjectKey{Namespace: machineContext.Machine.GetNamespace(), Name: *machineBootstrapSecretReferenceName + "-userdata"}
-		infraClusterClient, _, err := infraClusterMock.GenerateInfraClusterClient(clusterContext)
+		infraClusterClient, _, err := infraClusterMock.GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context)
 		Expect(err).NotTo(HaveOccurred())
 		bootstrapDataSecret := &corev1.Secret{}
 		err = infraClusterClient.Get(gocontext.Background(), machineBootstrapSecretReferenceKey, bootstrapDataSecret)
@@ -350,8 +349,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 		setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-		clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-		infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil).Times(2)
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil).Times(2)
 
 		out, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 
@@ -368,7 +366,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 		machineBootstrapSecretReferenceName := machineContext.Machine.Spec.Bootstrap.DataSecretName
 		machineBootstrapSecretReferenceKey := client.ObjectKey{Namespace: machineContext.Machine.GetNamespace(), Name: *machineBootstrapSecretReferenceName + "-userdata"}
-		infraClusterClient, _, err := infraClusterMock.GenerateInfraClusterClient(clusterContext)
+		infraClusterClient, _, err := infraClusterMock.GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context)
 		Expect(err).NotTo(HaveOccurred())
 
 		bootstrapDataSecret := &corev1.Secret{}
@@ -378,6 +376,38 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 		Expect(bootstrapUserDataSecret.Data["userdata"]).To(Equal([]byte("shell-script")))
 	})
 
+	It("should be able to delete KubeVirt VM even when cluster objects don't exist", func() {
+		controllerutil.AddFinalizer(kubevirtMachine, infrav1.MachineFinalizer)
+		objects := []client.Object{
+			machine,
+			kubevirtMachine,
+			bootstrapUserDataSecret,
+			vm,
+		}
+
+		setupClient(machineFactoryMock, objects)
+
+		machineContext = &context.MachineContext{
+			Context:         gocontext.Background(),
+			Machine:         machine,
+			KubevirtMachine: kubevirtMachine,
+			Logger:          testLogger,
+		}
+
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
+
+		out, err := kubevirtMachineReconciler.reconcileDelete(machineContext)
+		Expect(err).ShouldNot(HaveOccurred())
+		Expect(out).To(Equal(ctrl.Result{}))
+
+		// vm should be deleted
+		vmKey := client.ObjectKey{Namespace: kubevirtMachine.Namespace, Name: kubevirtMachine.Name}
+		err = fakeClient.Get(gocontext.Background(), vmKey, vm)
+		Expect(err).To(HaveOccurred())
+
+		Expect(machineContext.Machine.ObjectMeta.Finalizers).To(HaveLen(0))
+	})
+
 	It("should create KubeVirt VM with externally managed cluster and no ssh key", func() {
 
 		kubevirtCluster.Annotations = map[string]string{
@@ -395,8 +425,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 		setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-		clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-		infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 		out, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 
@@ -433,8 +462,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 		setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-		clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-		infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 		out, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 
@@ -482,8 +510,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 		setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-		clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-		infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 		Expect(machineContext.KubevirtMachine.Status.Ready).To(BeFalse())
 		out, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
@@ -519,8 +546,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 			setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-			clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-			infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+			infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 			// kubevirtMachineReconciler.Client
 			kubevirtMachineKey := types.NamespacedName{Namespace: kubevirtMachine.Namespace, Name: kubevirtMachine.Name}
 			_, err := kubevirtMachineReconciler.Reconcile(machineContext, ctrl.Request{NamespacedName: kubevirtMachineKey})
@@ -549,8 +575,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 				setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-				clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-				infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+				infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 				// kubevirtMachineReconciler.Client
 				kubevirtMachineKey := types.NamespacedName{Namespace: kubevirtMachine.Namespace, Name: kubevirtMachine.Name}
 				_, err := kubevirtMachineReconciler.reconcileDelete(machineContext)
@@ -584,8 +609,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 				setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-				clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-				infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+				infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 				_, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 
@@ -612,8 +636,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 				setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-				clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-				infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+				infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 				_, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 
@@ -632,10 +655,9 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 					sshKeySecret,
 				}
 
-				// clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
 				setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-				infraClusterMock.EXPECT().GenerateInfraClusterClient(gomock.Any()).Return(fakeClient, cluster.Namespace, nil)
+				infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 				_, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 				Expect(err).ShouldNot(BeNil())
@@ -672,8 +694,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 				machineMock.EXPECT().SupportsCheckingIsBootstrapped().Return(false).Times(1)
 				machineFactoryMock.EXPECT().NewMachine(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(machineMock, nil).Times(1)
 
-				clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-				infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+				infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 				_, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 				Expect(err).ShouldNot(HaveOccurred())
@@ -720,8 +741,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 				setupClient(machineFactoryMock, objects)
 
-				clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-				infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+				infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 				_, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 				Expect(err).ShouldNot(HaveOccurred())
@@ -770,8 +790,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 				setupClient(machineFactoryMock, objects)
 
-				clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-				infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+				infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 				_, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
 				Expect(err).ShouldNot(HaveOccurred())
@@ -806,8 +825,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 		setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-		clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-		infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 		Expect(machineContext.KubevirtMachine.Status.Ready).To(BeTrue())
 		out, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
@@ -830,8 +848,7 @@ var _ = Describe("reconcile a kubevirt machine", func() {
 
 		setupClient(kubevirt.DefaultMachineFactory{}, objects)
 
-		clusterContext := &context.ClusterContext{Context: machineContext.Context, Cluster: machineContext.Cluster, KubevirtCluster: machineContext.KubevirtCluster, Logger: machineContext.Logger}
-		infraClusterMock.EXPECT().GenerateInfraClusterClient(clusterContext).Return(fakeClient, cluster.Namespace, nil)
+		infraClusterMock.EXPECT().GenerateInfraClusterClient(kubevirtMachine.Spec.InfraClusterSecretRef, kubevirtMachine.Namespace, machineContext.Context).Return(fakeClient, kubevirtMachine.Namespace, nil)
 
 		Expect(machineContext.KubevirtMachine.Status.Ready).To(BeTrue())
 		out, err := kubevirtMachineReconciler.reconcileNormal(machineContext)
diff --git a/e2e-tests/common.go b/e2e-tests/common.go
index 4539dd4..6c2a380 100644
--- a/e2e-tests/common.go
+++ b/e2e-tests/common.go
@@ -13,7 +13,8 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 )
 
-func DeleteAndWait(k8sclient client.Client, obj client.Object, key client.ObjectKey, timeoutSeconds uint) {
+func DeleteAndWait(k8sclient client.Client, obj client.Object, timeoutSeconds uint) {
+	key := client.ObjectKeyFromObject(obj)
 	Eventually(func() error {
 		err := k8sclient.Get(context.Background(), key, obj)
 		if err != nil && k8serrors.IsNotFound(err) {
@@ -22,9 +23,11 @@ func DeleteAndWait(k8sclient client.Client, obj client.Object, key client.Object
 			return err
 		}
 
-		err = k8sclient.Delete(context.Background(), obj)
-		if err != nil {
-			return err
+		if obj.GetDeletionTimestamp().IsZero() {
+			err = k8sclient.Delete(context.Background(), obj)
+			if err != nil {
+				return err
+			}
 		}
 		return fmt.Errorf("waiting on object %s to be deleted", key)
 	}, time.Duration(timeoutSeconds)*time.Second, 1*time.Second).Should(BeNil())
diff --git a/e2e-tests/create-cluster_test.go b/e2e-tests/create-cluster_test.go
index 8b5ea2a..e15d20c 100644
--- a/e2e-tests/create-cluster_test.go
+++ b/e2e-tests/create-cluster_test.go
@@ -76,36 +76,21 @@ var _ = Describe("CreateCluster", func() {
 					Name: namespace,
 				},
 			}
-			_ = k8sclient.Delete(context.Background(), ns)
-			_ = os.RemoveAll(tmpDir)
-		}()
 
-		// Typically the machine deployment and machines should not need to get removed before
-		// the Cluster object. However we have a bug today that prevents proper tear down
-		// of the cluster unless the machines are removed first.
-		//
-		// Tracking this issue here: https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/issues/65
-		// TODO remove the logic that delets the machine and kubevirt machines once this issue is resolved.
-		By("removing machine deployment")
-		machineDeployment := &clusterv1.MachineDeployment{}
-		key := client.ObjectKey{Namespace: namespace, Name: "kvcluster-md-0"}
-		tests.DeleteAndWait(k8sclient, machineDeployment, key, 120)
-
-		By("removing all kubevirt machines")
-		machineList := &infrav1.KubevirtMachineList{}
-		err := k8sclient.List(context.Background(), machineList, client.InNamespace(namespace))
-		Expect(err).ToNot(HaveOccurred())
+			tests.DeleteAndWait(k8sclient, ns, 120)
 
-		for _, machine := range machineList.Items {
-			key := client.ObjectKey{Namespace: namespace, Name: machine.Name}
-			tests.DeleteAndWait(k8sclient, &machine, key, 120)
-		}
+		}()
 
-		By("removing cluster")
-		cluster := &clusterv1.Cluster{}
-		key = client.ObjectKey{Namespace: namespace, Name: "kvcluster"}
-		tests.DeleteAndWait(k8sclient, cluster, key, 120)
+		_ = os.RemoveAll(tmpDir)
 
+		By("removing cluster")
+		cluster := &clusterv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				Namespace: namespace,
+				Name:      "kvcluster",
+			},
+		}
+		tests.DeleteAndWait(k8sclient, cluster, 120)
 	})
 
 	waitForBootstrappedMachines := func() {
@@ -362,6 +347,15 @@ var _ = Describe("CreateCluster", func() {
 
 		By("Waiting for all tenant nodes to get provider id")
 		waitForNodeUpdate()
+
+		By("Ensuring cluster teardown works without race conditions by deleting namespace")
+		ns := &corev1.Namespace{
+			ObjectMeta: metav1.ObjectMeta{
+				Name: namespace,
+			},
+		}
+		tests.DeleteAndWait(k8sclient, ns, 120)
+
 	})
 
 	It("creating a simple cluster with persistent VMs", func() {