Repository: rancher/fleet

Issue:
[SURE-10589] Fleet does not delete the "cluster-fleet-local-<clusterName>-<hash>" namespace when a cluster is deleted from Rancher UI.
## SURE-10589

### Issue description:
Deleting a cluster from Rancher UI does not clean up the `cluster-fleet-local-<clusterName>-<hash>` namespace from the `SYSTEM` project which is created by fleet. 

### Business impact:
There is no business impact as such but customer has to delete the namespace manually. 

### Troubleshooting steps:
We can clean up this namespace manually or restart the fleet-controller which would clean up the namespace.

### Repro steps:

Create a Rancher provisioned downstream cluster.
This would create a namespace in the SYSTEM project for this cluster in `cluster-fleet-local-<clusterName>-<hash>` format. 
Delete the cluster from Rancher UI. 
This should ideally trigger fleet to remove this namespace once the cluster is deleted. However, the namespace remains as an orphan resource. 

### Workaround:
Is a workaround available and implemented? yes
What is the workaround: Restart the fleet-controller to resolve the issue. 

### Actual behavior:
Upon cluster deletion the `cluster-fleet-local-<clusterName>-<hash>` namespace in SYSTEM project is not cleaned up automatically. 

### Expected behavior:
Once the cluster is deleted, all of its associated resources should be cleaned up from the cluster. 

### Files, logs, traces:
Logs can be uploaded to the Jira on demand since this is easily reproducible.


Hints from issue comments:
@dharmit: @kkaempf can I work on this one? I haven't worked on anything related to Fleet yet, so would like to get my hands dirty if that's OK.@kkaempf: It's yours, good luck 😉 @kkaempf: @dharmit please post your updates here (or file a pull request, referencing this issue 😉 )@dharmit: I'm still looking into this trying to understand the flow of things. One thing I have been able to deduce so far is that namespace doesn't get deleted for older version too - I tried 2.10.8.

Beyond that, I'm looking for what could be the right place to make this change because a `clusters.fleet.cattle.io` resource seems to be created/owned by a `clusters.provisioning.cattle.io`. My mind doesn't have a full picture of code/architecture of fleet, so I'll keep posting my thoughts/learning here as things become clearer.

@dharmit: Using the scripts `setup-single-cluster` in `dev` directory, I managed to create a development setup for fleet that doesn't involve/install Rancher. Then, I scaled down the `fleet-controller` deployment to zero replicas and ran the `fleet-controller`, `fleet-agentmanagement` and `fleet-cleanup` containers through IDE. 

After inserting some breakpoints in the file `internal/cmd/controller/agentmanagement/controllers/cluster/controller.go` I expected to hit them when a downstream cluster is created using `dev/setup-k3ds-downstream` script, but that didn't happen. Maybe I'm expecting it wrong. Downstream cluster created this way doesn't seem to be creating a namespace in the dev setup either.

Maybe naive, but I added a `clusters.OnRemove` line after: https://github.com/rancher/fleet/blob/ed1d72b2f0282c627074eb435b0eae2c21815ba2/internal/cmd/controller/agentmanagement/controllers/cluster/controller.go#L54

but that didn't seem to be getting triggered upon removal of a downstream cluster.

If you have any pointers as to how to approach this one, let me know. :)
cc @manno @weyfonk @0xavi0 @kkaempf.@sbulage: #### System Information
| Rancher Version | Fleet Version |
|--------|--------|
| v2.13-0e9916960a4a3e1ef0a0aa91968b3ebfd98c948d-head | 108.0.0+up0.14.0-beta.1 | 


### Verification Steps performed
See below video for detailed steps:

<details><summary>Creating and deleting EKS cluster via Rancher UI</summary>

https://github.com/user-attachments/assets/81b4d771-1987-41b9-b156-92ad0e9dbbe5

</details> 

Solution patch:
diff --git a/.github/scripts/run-integration-tests-group1.sh b/.github/scripts/run-integration-tests-group1.sh
index 48ab11be..293ad4b6 100755
--- a/.github/scripts/run-integration-tests-group1.sh
+++ b/.github/scripts/run-integration-tests-group1.sh
@@ -2,8 +2,8 @@
 
 set -euxo pipefail
 
-SETUP_ENVTEST_VER=${SETUP_ENVTEST_VER-v0.0.0-20250218120612-6f6111124902}
-ENVTEST_K8S_VERSION=${ENVTEST_K8S_VERSION-1.32}
+SETUP_ENVTEST_VER=${SETUP_ENVTEST_VER-v0.0.0-20251014082336-b8f11375258f}
+ENVTEST_K8S_VERSION=${ENVTEST_K8S_VERSION-1.34}
 
 # install and prepare setup-envtest
 go install sigs.k8s.io/controller-runtime/tools/setup-envtest@"$SETUP_ENVTEST_VER"
diff --git a/.github/scripts/run-integration-tests-group2.sh b/.github/scripts/run-integration-tests-group2.sh
index 9a085a1d..f523e7ae 100755
--- a/.github/scripts/run-integration-tests-group2.sh
+++ b/.github/scripts/run-integration-tests-group2.sh
@@ -2,8 +2,8 @@
 
 set -euxo pipefail
 
-SETUP_ENVTEST_VER=${SETUP_ENVTEST_VER-v0.0.0-20250218120612-6f6111124902}
-ENVTEST_K8S_VERSION=${ENVTEST_K8S_VERSION-1.32}
+SETUP_ENVTEST_VER=${SETUP_ENVTEST_VER-v0.0.0-20251014082336-b8f11375258f}
+ENVTEST_K8S_VERSION=${ENVTEST_K8S_VERSION-1.34}
 
 # install and prepare setup-envtest
 go install sigs.k8s.io/controller-runtime/tools/setup-envtest@"$SETUP_ENVTEST_VER"
@@ -15,5 +15,5 @@ find ./integrationtests -type d -not -path '*/.git*' \
   -not -path './integrationtests/agent*' \
   -not -path './integrationtests/bundlereader*' \
   -not -path './integrationtests/cli*' \
-  -not -path './integrationtests/controller*' | \
-  xargs ginkgo --github-output --trace
\ No newline at end of file
+  -not -path './integrationtests/controller*' -print0 | \
+  xargs -0 ginkgo --github-output --trace
\ No newline at end of file
diff --git a/.github/scripts/run-integration-tests.sh b/.github/scripts/run-integration-tests.sh
index e0dd7561..8989094f 100755
--- a/.github/scripts/run-integration-tests.sh
+++ b/.github/scripts/run-integration-tests.sh
@@ -2,8 +2,8 @@
 
 set -euxo pipefail
 
-SETUP_ENVTEST_VER=${SETUP_ENVTEST_VER-v0.0.0-20250218120612-6f6111124902}
-ENVTEST_K8S_VERSION=${ENVTEST_K8S_VERSION-1.32}
+SETUP_ENVTEST_VER=${SETUP_ENVTEST_VER-v0.0.0-20251014082336-b8f11375258f}
+ENVTEST_K8S_VERSION=${ENVTEST_K8S_VERSION-1.34}
 
 # install and prepare setup-envtest
 go install sigs.k8s.io/controller-runtime/tools/setup-envtest@"$SETUP_ENVTEST_VER"
diff --git a/.github/workflows/e2e-nightly-ci.yml b/.github/workflows/e2e-nightly-ci.yml
index 6d13e808..e6e5ed94 100644
--- a/.github/workflows/e2e-nightly-ci.yml
+++ b/.github/workflows/e2e-nightly-ci.yml
@@ -172,6 +172,9 @@ jobs:
           GIT_REPO_URL: "git@github.com:fleetrepoci/test.git"
           GIT_REPO_HOST: "github.com"
           GIT_REPO_USER: "git"
+          GITHUB_APP_ID:  ${{ secrets.CI_GITHUB_APP_ID }}
+          GITHUB_APP_INSTALLATION_ID:  ${{ secrets.CI_GITHUB_APP_INSTALLATION_ID }}
+          GITHUB_APP_PRIVATE_KEY: ${{ secrets.CI_GITHUB_APP_PRIVATE_KEY }}
           GITHUB_PAT_TOKEN: ${{ secrets.CI_PRIVATE_REPO_PAT }}
           GIT_REPO_BRANCH: ${{ matrix.k3s_version }}
           CI_OCI_USERNAME:  ${{ secrets.CI_OCI_USERNAME }}
diff --git a/.github/workflows/e2e-test-fleet-in-rancher.yml b/.github/workflows/e2e-test-fleet-in-rancher.yml
index a6c9abea..babf5512 100644
--- a/.github/workflows/e2e-test-fleet-in-rancher.yml
+++ b/.github/workflows/e2e-test-fleet-in-rancher.yml
@@ -227,7 +227,7 @@ jobs:
 
       -
         name: Checkout rancher/rancher
-        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5
         with:
           fetch-depth: 0
           repository: rancher/rancher
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index bc1069c8..e1dafff8 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -80,8 +80,8 @@ jobs:
         uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3
 
       - name: Install Cosign
-        uses: sigstore/cosign-installer@d7543c93d881b35a8faa02e8e3605f69b7a1ce62 # v3.10.0
-      - uses: rancherlabs/slsactl/actions/install-slsactl@b01dab9e871b6d039a1befb15a63175eccd90cbc # v0.0.15
+        uses: sigstore/cosign-installer@faadad0cce49287aee09b3a48701e75088a2c6ad # v4.0.0
+      - uses: rancherlabs/slsactl/actions/install-slsactl@883774aee17dd2d82f388f3e93a8df623c3437a6 # v0.0.16
 
       - name: "Read Vault Secrets"
         uses: rancher-eio/read-vault-secrets@main
diff --git a/internal/cmd/agent/deployer/internal/diff/diff.go b/internal/cmd/agent/deployer/internal/diff/diff.go
index 373df561..fe6cacfc 100644
--- a/internal/cmd/agent/deployer/internal/diff/diff.go
+++ b/internal/cmd/agent/deployer/internal/diff/diff.go
@@ -11,10 +11,12 @@ import (
 	"errors"
 	"fmt"
 	"reflect"
+	"sort"
 
 	jsonpatch "github.com/evanphx/json-patch"
 
 	corev1 "k8s.io/api/core/v1"
+	discoveryv1 "k8s.io/api/discovery/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/util/jsonmergepatch"
@@ -423,6 +425,8 @@ func Normalize(un *unstructured.Unstructured, opts ...Option) {
 		normalizeRole(un, o)
 	} else if gvk.Group == "" && gvk.Kind == "Endpoints" {
 		normalizeEndpoint(un, o)
+	} else if gvk.Group == "discovery.k8s.io" && gvk.Kind == "EndpointSlice" {
+		normalizeEndpointSlice(un, o)
 	}
 
 	err := o.normalizer.Normalize(un)
@@ -477,7 +481,9 @@ func NormalizeSecret(un *unstructured.Unstructured, opts ...Option) {
 	}
 }
 
-// normalizeEndpoint normalizes endpoint meaning that EndpointSubsets are sorted lexicographically
+// normalizeEndpoint normalizes endpoint meaning that EndpointSubsets are sorted lexicographically.
+// Note: The Endpoints API (core/v1) is deprecated in favor of EndpointSlice (discovery.k8s.io/v1).
+// Endpoints are supported for backward compatibility as users may still deploy them.
 func normalizeEndpoint(un *unstructured.Unstructured, o options) {
 	if un == nil {
 		return
@@ -486,7 +492,7 @@ func normalizeEndpoint(un *unstructured.Unstructured, o options) {
 	if gvk.Group != "" || gvk.Kind != "Endpoints" {
 		return
 	}
-	// nolint: staticcheck // Endpoint is deprecated; see fleet#3760.
+	// nolint: staticcheck // Endpoints is deprecated but still supported; see fleet#3760.
 	var ep corev1.Endpoints
 	err := runtime.DefaultUnstructuredConverter.FromUnstructured(un.Object, &ep)
 	if err != nil {
@@ -515,6 +521,82 @@ func normalizeEndpoint(un *unstructured.Unstructured, o options) {
 	un.Object = newObj
 }
 
+// normalizeEndpointSlice normalizes EndpointSlice by sorting endpoints and ports lexicographically
+// and setting default protocols. EndpointSlice is the successor to the deprecated Endpoints API.
+func normalizeEndpointSlice(un *unstructured.Unstructured, o options) {
+	if un == nil {
+		return
+	}
+	gvk := un.GroupVersionKind()
+	if gvk.Group != "discovery.k8s.io" || gvk.Kind != "EndpointSlice" {
+		return
+	}
+
+	var eps discoveryv1.EndpointSlice
+	err := runtime.DefaultUnstructuredConverter.FromUnstructured(un.Object, &eps)
+	if err != nil {
+		o.log.Error(err, "Failed to convert from unstructured into EndpointSlice")
+		return
+	}
+
+	// add default protocol to ports if it is empty
+	for p, port := range eps.Ports {
+		if port.Protocol == nil {
+			tcp := corev1.ProtocolTCP
+			port.Protocol = &tcp
+			eps.Ports[p] = port
+		}
+	}
+
+	// Sort endpoints by their addresses for consistent comparison
+	sortEndpointSliceEndpoints(eps.Endpoints)
+	sortEndpointSlicePorts(eps.Ports)
+
+	newObj, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&eps)
+	if err != nil {
+		o.log.Info(fmt.Sprintf(couldNotMarshalErrMsg, gvk, err))
+		return
+	}
+	un.Object = newObj
+}
+
+// sortEndpointSliceEndpoints sorts EndpointSlice endpoints by their addresses
+func sortEndpointSliceEndpoints(endpoints []discoveryv1.Endpoint) {
+	sort.Slice(endpoints, func(i, j int) bool {
+		// Sort by first address if available
+		if len(endpoints[i].Addresses) > 0 && len(endpoints[j].Addresses) > 0 {
+			addrCmp := bytes.Compare([]byte(endpoints[i].Addresses[0]), []byte(endpoints[j].Addresses[0]))
+			if addrCmp != 0 {
+				return addrCmp < 0
+			}
+		}
+		// Sort by target ref if available
+		if endpoints[i].TargetRef != nil && endpoints[j].TargetRef != nil {
+			return endpoints[i].TargetRef.UID < endpoints[j].TargetRef.UID
+		}
+		return false
+	})
+}
+
+// sortEndpointSlicePorts sorts EndpointSlice ports by port number and protocol
+func sortEndpointSlicePorts(ports []discoveryv1.EndpointPort) {
+	sort.Slice(ports, func(i, j int) bool {
+		// Compare by port number first
+		if ports[i].Port != nil && ports[j].Port != nil && *ports[i].Port != *ports[j].Port {
+			return *ports[i].Port < *ports[j].Port
+		}
+		// Then by name
+		if ports[i].Name != nil && ports[j].Name != nil && *ports[i].Name != *ports[j].Name {
+			return *ports[i].Name < *ports[j].Name
+		}
+		// Then by protocol
+		if ports[i].Protocol != nil && ports[j].Protocol != nil {
+			return *ports[i].Protocol < *ports[j].Protocol
+		}
+		return false
+	})
+}
+
 // normalizeRole mutates the supplied Role/ClusterRole and sets rules to null if it is an empty list or an aggregated role
 func normalizeRole(un *unstructured.Unstructured, o options) {
 	if un == nil {
diff --git a/internal/cmd/agent/deployer/internal/diff/kubernetes_vendor/pkg/api/v1/endpoints/util.go b/internal/cmd/agent/deployer/internal/diff/kubernetes_vendor/pkg/api/v1/endpoints/util.go
index 0effcb7d..dfcaff03 100644
--- a/internal/cmd/agent/deployer/internal/diff/kubernetes_vendor/pkg/api/v1/endpoints/util.go
+++ b/internal/cmd/agent/deployer/internal/diff/kubernetes_vendor/pkg/api/v1/endpoints/util.go
@@ -43,7 +43,9 @@ func LessEndpointAddress(a, b *v1.EndpointAddress) bool {
 
 // SortSubsets sorts an array of EndpointSubset objects in place.  For ease of
 // use it returns the input slice.
-// nolint: staticcheck // EndpointSubset is deprecated; see fleet#3760.
+// Note: EndpointSubset is part of the deprecated Endpoints API (core/v1).
+// The successor is EndpointSlice (discovery.k8s.io/v1).
+// nolint: staticcheck // EndpointSubset is deprecated but still supported; see fleet#3760.
 func SortSubsets(subsets []v1.EndpointSubset) []v1.EndpointSubset {
 	for i := range subsets {
 		ss := &subsets[i]
@@ -60,7 +62,7 @@ func hashObject(hasher hash.Hash, obj interface{}) []byte {
 	return hasher.Sum(nil)
 }
 
-// nolint: staticcheck // EndpointSubset is deprecated; see fleet#3760.
+// nolint: staticcheck // EndpointSubset is deprecated but still supported; see fleet#3760.
 type subsetsByHash []v1.EndpointSubset
 
 func (sl subsetsByHash) Len() int      { return len(sl) }
diff --git a/internal/cmd/agent/deployer/normalizers/mutatingwebhook.go b/internal/cmd/agent/deployer/normalizers/mutatingwebhook.go
index 6368656d..164e14a1 100644
--- a/internal/cmd/agent/deployer/normalizers/mutatingwebhook.go
+++ b/internal/cmd/agent/deployer/normalizers/mutatingwebhook.go
@@ -6,7 +6,6 @@ import (
 	"github.com/rancher/fleet/internal/cmd/agent/deployer/objectset"
 
 	adregv1 "k8s.io/api/admissionregistration/v1"
-	adregv1beta1 "k8s.io/api/admissionregistration/v1beta1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/runtime/schema"
@@ -25,37 +24,9 @@ func (m *MutatingWebhookNormalizer) Normalize(un *unstructured.Unstructured) err
 		return nil
 	}
 
-	if gvk.Version == "v1beta1" {
-		return m.convertMutatingWebhookV1beta1(un)
-	}
-
 	return m.convertMutatingWebhookV1(un)
 }
 
-func (m *MutatingWebhookNormalizer) convertMutatingWebhookV1beta1(un *unstructured.Unstructured) error {
-	var webhook adregv1beta1.MutatingWebhookConfiguration
-	err := runtime.DefaultUnstructuredConverter.FromUnstructured(un.Object, &webhook)
-	if err != nil {
-		logrus.Error("Failed to convert unstructured to webhook")
-		return nil
-	}
-
-	for i, config := range webhook.Webhooks {
-		if webhook.UID == "" && string(config.ClientConfig.CABundle) == "\n" {
-			live := lookupLive(un.GroupVersionKind(), un.GetName(), un.GetNamespace(), m.Live)
-			liveWebhook, ok := live.(*unstructured.Unstructured)
-			if !ok {
-				continue
-			}
-			if err := setMutatingWebhookV1beta1CacertNil(liveWebhook, i); err != nil {
-				logrus.Errorf("Failed to normalize webhook cacert, err: %v", err)
-				return nil
-			}
-		}
-	}
-	return nil
-}
-
 func (m *MutatingWebhookNormalizer) convertMutatingWebhookV1(un *unstructured.Unstructured) error {
 	var webhook adregv1.MutatingWebhookConfiguration
 	err := runtime.DefaultUnstructuredConverter.FromUnstructured(un.Object, &webhook)
@@ -106,32 +77,6 @@ func setMutatingWebhookV1CacertNil(un *unstructured.Unstructured, index int) err
 	return nil
 }
 
-func setMutatingWebhookV1beta1CacertNil(un *unstructured.Unstructured, index int) error {
-	var webhook adregv1beta1.MutatingWebhookConfiguration
-	err := runtime.DefaultUnstructuredConverter.FromUnstructured(un.Object, &webhook)
-	if err != nil {
-		logrus.Error("Failed to convert unstructured to webhook")
-		return err
-	}
-
-	if index >= len(webhook.Webhooks) {
-		return nil
-	}
-	webhook.Webhooks[index].ClientConfig.CABundle = nil
-	newObj, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&webhook)
-	if err != nil {
-		logrus.Errorf("Failed to convert unstructured to webhook, err: %v", err)
-		return err
-	}
-	if webhook.Webhooks != nil {
-		if err = unstructured.SetNestedField(un.Object, newObj["webhooks"], "webhooks"); err != nil {
-			logrus.Errorf("MutatingWebhook normalization error: %v", err)
-			return err
-		}
-	}
-	return nil
-}
-
 func lookupLive(gvk schema.GroupVersionKind, name, namespace string, live objectset.ObjectByGVK) runtime.Object {
 	key := objectset.ObjectKey{
 		Namespace: namespace,
diff --git a/internal/cmd/agent/deployer/normalizers/validatingwebhook.go b/internal/cmd/agent/deployer/normalizers/validatingwebhook.go
index bd106979..6febe802 100644
--- a/internal/cmd/agent/deployer/normalizers/validatingwebhook.go
+++ b/internal/cmd/agent/deployer/normalizers/validatingwebhook.go
@@ -6,7 +6,6 @@ import (
 	"github.com/rancher/fleet/internal/cmd/agent/deployer/objectset"
 
 	adregv1 "k8s.io/api/admissionregistration/v1"
-	adregv1beta1 "k8s.io/api/admissionregistration/v1beta1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/runtime"
 )
@@ -24,37 +23,9 @@ func (v *ValidatingWebhookNormalizer) Normalize(un *unstructured.Unstructured) e
 		return nil
 	}
 
-	if gvk.Version == "v1beta1" {
-		return v.convertValidatingWebhookV1beta1(un)
-	}
-
 	return v.convertValidatingWebhookV1(un)
 }
 
-func (v *ValidatingWebhookNormalizer) convertValidatingWebhookV1beta1(un *unstructured.Unstructured) error {
-	var webhook adregv1beta1.ValidatingWebhookConfiguration
-	err := runtime.DefaultUnstructuredConverter.FromUnstructured(un.Object, &webhook)
-	if err != nil {
-		logrus.Error("Failed to convert unstructured to webhook")
-		return nil
-	}
-
-	for i, config := range webhook.Webhooks {
-		if webhook.UID == "" && string(config.ClientConfig.CABundle) == "\n" {
-			live := lookupLive(un.GroupVersionKind(), un.GetName(), un.GetNamespace(), v.Live)
-			liveWebhook, ok := live.(*unstructured.Unstructured)
-			if !ok {
-				continue
-			}
-			if err := setValidatingWebhookV1beta1CacertNil(liveWebhook, i); err != nil {
-				logrus.Errorf("Failed to normalize webhook cacert, err: %v", err)
-				return nil
-			}
-		}
-	}
-	return nil
-}
-
 func (v *ValidatingWebhookNormalizer) convertValidatingWebhookV1(un *unstructured.Unstructured) error {
 	var webhook adregv1.ValidatingWebhookConfiguration
 	err := runtime.DefaultUnstructuredConverter.FromUnstructured(un.Object, &webhook)
@@ -104,29 +75,3 @@ func setValidatingWebhookV1CacertNil(un *unstructured.Unstructured, index int) e
 	}
 	return nil
 }
-
-func setValidatingWebhookV1beta1CacertNil(un *unstructured.Unstructured, index int) error {
-	var webhook adregv1beta1.ValidatingWebhookConfiguration
-	err := runtime.DefaultUnstructuredConverter.FromUnstructured(un.Object, &webhook)
-	if err != nil {
-		logrus.Error("Failed to convert unstructured to webhook")
-		return err
-	}
-
-	if index >= len(webhook.Webhooks) {
-		return nil
-	}
-	webhook.Webhooks[index].ClientConfig.CABundle = nil
-	newObj, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&webhook)
-	if err != nil {
-		logrus.Errorf("Failed to convert unstructured to webhook, err: %v", err)
-		return err
-	}
-	if webhook.Webhooks != nil {
-		if err = unstructured.SetNestedField(un.Object, newObj["webhooks"], "webhooks"); err != nil {
-			logrus.Errorf("ValidatingWebhook normalization error: %v", err)
-			return err
-		}
-	}
-	return nil
-}
diff --git a/internal/cmd/controller/finalize/finalize.go b/internal/cmd/controller/finalize/finalize.go
index 1f89c4fe..a1424c64 100644
--- a/internal/cmd/controller/finalize/finalize.go
+++ b/internal/cmd/controller/finalize/finalize.go
@@ -22,6 +22,7 @@ const (
 	GitRepoFinalizer          = "fleet.cattle.io/gitrepo-finalizer"
 	BundleFinalizer           = "fleet.cattle.io/bundle-finalizer"
 	BundleDeploymentFinalizer = "fleet.cattle.io/bundle-deployment-finalizer"
+	ClusterFinalizer          = "fleet.cattle.io/cluster-finalizer"
 )
 
 // PurgeBundles deletes all bundles related to the given resource namespaced name
diff --git a/internal/cmd/controller/reconciler/cluster_controller.go b/internal/cmd/controller/reconciler/cluster_controller.go
index b692375b..875dacd8 100644
--- a/internal/cmd/controller/reconciler/cluster_controller.go
+++ b/internal/cmd/controller/reconciler/cluster_controller.go
@@ -10,6 +10,7 @@ import (
 	"sort"
 	"time"
 
+	"github.com/rancher/fleet/internal/cmd/controller/finalize"
 	"github.com/rancher/fleet/internal/cmd/controller/summary"
 	"github.com/rancher/fleet/internal/metrics"
 	"github.com/rancher/fleet/internal/resourcestatus"
@@ -33,6 +34,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/builder"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
+	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/event"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
 	"sigs.k8s.io/controller-runtime/pkg/log"
@@ -124,11 +126,18 @@ func (r *ClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 
 	cluster := &fleet.Cluster{}
 	err := r.Get(ctx, req.NamespacedName, cluster)
-	if apierrors.IsNotFound(err) {
-		metrics.ClusterCollector.Delete(req.Name, req.Namespace)
-		return ctrl.Result{}, nil
-	} else if err != nil {
-		return ctrl.Result{}, err
+	if err != nil {
+		return ctrl.Result{}, client.IgnoreNotFound(err)
+	}
+
+	if !cluster.DeletionTimestamp.IsZero() {
+		return r.handleDelete(ctx, cluster)
+	}
+
+	if err := r.ensureFinalizer(ctx, cluster); err != nil {
+		// Retry without updating the status, as this should be a transient error about which users can't do
+		// anything.
+		return ctrl.Result{}, fmt.Errorf("%w, failed to add finalizer to bundle: %w", fleetutil.ErrRetryable, err)
 	}
 
 	if cluster.Status.Namespace == "" {
@@ -337,3 +346,42 @@ func (r *ClusterReconciler) mapBundleDeploymentToCluster(ctx context.Context, a
 		},
 	}}
 }
+
+// ensureFinalizer adds a finalizer to a recently created cluster.
+func (r *ClusterReconciler) ensureFinalizer(ctx context.Context, cluster *fleet.Cluster) error {
+	if controllerutil.ContainsFinalizer(cluster, finalize.ClusterFinalizer) {
+
+		return nil
+	}
+
+	controllerutil.AddFinalizer(cluster, finalize.ClusterFinalizer)
+
+	return r.Update(ctx, cluster)
+}
+
+// handleDelete runs cleanup the namespace associated to a Cluster,
+// finally removing the finalizer to unblock the deletion of the object from kubernetes.
+func (r *ClusterReconciler) handleDelete(ctx context.Context, cluster *fleet.Cluster) (ctrl.Result, error) {
+	if !controllerutil.ContainsFinalizer(cluster, finalize.ClusterFinalizer) {
+
+		return ctrl.Result{}, nil
+	}
+
+	if cluster.Status.Namespace != "" {
+		// pro-actively delete the cluster's namespace.
+		ns := &corev1.Namespace{
+			ObjectMeta: metav1.ObjectMeta{
+				Name: cluster.Status.Namespace,
+			},
+		}
+		if err := r.Delete(ctx, ns); err != nil && !apierrors.IsNotFound(err) {
+
+			return ctrl.Result{}, err
+		}
+	}
+
+	metrics.ClusterCollector.Delete(cluster.Name, cluster.Namespace)
+	controllerutil.RemoveFinalizer(cluster, finalize.ClusterFinalizer)
+
+	return ctrl.Result{}, r.Update(ctx, cluster)
+}



Test patch:
diff --git a/integrationtests/utils/helpers.go b/integrationtests/utils/helpers.go
index d6657cb0..125d0d21 100644
--- a/integrationtests/utils/helpers.go
+++ b/integrationtests/utils/helpers.go
@@ -7,6 +7,8 @@ import (
 	"github.com/rancher/fleet/pkg/apis/fleet.cattle.io/v1alpha1"
 
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/util/retry"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 )
 
@@ -48,8 +50,20 @@ func CreateCluster(ctx context.Context, k8sClient client.Client, name, controlle
 	if err != nil {
 		return nil, err
 	}
-	cluster.Status.Namespace = clusterNs
-	err = k8sClient.Status().Update(ctx, cluster)
+
+	ns := types.NamespacedName{
+		Namespace: controllerNs,
+		Name:      name,
+	}
+	err = retry.RetryOnConflict(retry.DefaultBackoff, func() error {
+		c := &v1alpha1.Cluster{}
+		err := k8sClient.Get(ctx, ns, c)
+		if err != nil {
+			return err
+		}
+		c.Status.Namespace = clusterNs
+		return k8sClient.Status().Update(ctx, c)
+	})
 	return cluster, err
 }
 
diff --git a/internal/cmd/controller/reconciler/cluster_controller_test.go b/internal/cmd/controller/reconciler/cluster_controller_test.go
new file mode 100644
index 00000000..680628a2
--- /dev/null
+++ b/internal/cmd/controller/reconciler/cluster_controller_test.go
@@ -0,0 +1,169 @@
+package reconciler
+
+import (
+	"context"
+	"time"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"github.com/rancher/fleet/internal/cmd/controller/finalize"
+	fleet "github.com/rancher/fleet/pkg/apis/fleet.cattle.io/v1alpha1"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/kubernetes/scheme"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+)
+
+type FakeQuery struct {
+}
+
+// BundlesForCluster returns empty list, so no cleanup is needed
+func (q *FakeQuery) BundlesForCluster(context.Context, *fleet.Cluster) ([]*fleet.Bundle, []*fleet.Bundle, error) {
+	return nil, nil, nil
+}
+
+var _ = Describe("ClusterReconciler", func() {
+	var (
+		ctx        context.Context
+		reconciler *ClusterReconciler
+		k8sclient  client.Client
+		cluster    *fleet.Cluster
+		req        reconcile.Request
+		sch        *runtime.Scheme
+	)
+
+	BeforeEach(func() {
+		ctx = context.Background()
+		sch = scheme.Scheme
+		Expect(fleet.AddToScheme(sch)).To(Succeed())
+		Expect(corev1.AddToScheme(sch)).To(Succeed())
+
+		cluster = &fleet.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-cluster",
+				Namespace: "fleet-local",
+			},
+			Status: fleet.ClusterStatus{
+				Namespace: "cluster-test-cluster-somehash",
+			},
+		}
+
+		req = reconcile.Request{
+			NamespacedName: types.NamespacedName{
+				Name:      "test-cluster",
+				Namespace: "fleet-local",
+			},
+		}
+	})
+
+	JustBeforeEach(func() {
+		if k8sclient == nil {
+			k8sclient = fake.NewClientBuilder().
+				WithScheme(sch).
+				WithObjects(cluster).
+				WithStatusSubresource(&fleet.Cluster{}).
+				Build()
+		}
+
+		reconciler = &ClusterReconciler{
+			Client: k8sclient,
+			Scheme: sch,
+			Query:  &FakeQuery{},
+		}
+	})
+
+	AfterEach(func() {
+		k8sclient = nil
+	})
+
+	Context("Reconcile finalizer", func() {
+		It("should add a finalizer to a new cluster", func() {
+			_, err := reconciler.Reconcile(ctx, req)
+			Expect(err).NotTo(HaveOccurred())
+
+			updatedCluster := &fleet.Cluster{}
+			err = k8sclient.Get(ctx, req.NamespacedName, updatedCluster)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(updatedCluster.Finalizers).To(ContainElement(finalize.ClusterFinalizer))
+		})
+	})
+
+	Context("Reconcile deletion", func() {
+		var clusterNamespace *corev1.Namespace
+
+		BeforeEach(func() {
+			cluster.Finalizers = []string{finalize.ClusterFinalizer}
+			now := metav1.NewTime(time.Now())
+			cluster.DeletionTimestamp = &now
+
+			clusterNamespace = &corev1.Namespace{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: cluster.Status.Namespace,
+				},
+			}
+		})
+
+		JustBeforeEach(func() {
+			k8sclient = fake.NewClientBuilder().
+				WithScheme(sch).
+				WithObjects(cluster, clusterNamespace).
+				WithStatusSubresource(&fleet.Cluster{}).
+				Build()
+
+			reconciler.Client = k8sclient
+		})
+
+		It("should delete the cluster namespace and remove the finalizer", func() {
+			_, err := reconciler.Reconcile(ctx, req)
+			Expect(err).NotTo(HaveOccurred())
+
+			// Check finalizer is removed
+			err = k8sclient.Get(ctx, req.NamespacedName, cluster)
+			Expect(err).To(HaveOccurred())
+			Expect(errors.IsNotFound(err)).To(BeTrue(), "cluster should be gone as finalizer is removed")
+
+			// Check namespace is deleted
+			ns := &corev1.Namespace{}
+			err = k8sclient.Get(ctx, client.ObjectKey{Name: clusterNamespace.Name}, ns)
+			Expect(err).To(HaveOccurred())
+			Expect(errors.IsNotFound(err)).To(BeTrue(), "cluster namespace should be deleted")
+		})
+
+		It("should remove the finalizer when cluster namespace is not set", func() {
+			// Remove cluster namespace before test
+			cluster.Status.Namespace = ""
+			k8sclient = fake.NewClientBuilder().
+				WithScheme(sch).
+				WithObjects(cluster).
+				WithStatusSubresource(&fleet.Cluster{}).
+				Build()
+			reconciler.Client = k8sclient
+
+			_, err := reconciler.Reconcile(ctx, req)
+			Expect(err).NotTo(HaveOccurred())
+
+			// Check finalizer is removed
+			err = k8sclient.Get(ctx, req.NamespacedName, cluster)
+			Expect(err).To(HaveOccurred())
+			Expect(errors.IsNotFound(err)).To(BeTrue(), "cluster should be gone as finalizer is removed")
+		})
+
+		It("should remove the finalizer even if the namespace is already gone", func() {
+			// Delete the namespace before the test
+			Expect(k8sclient.Delete(ctx, clusterNamespace)).To(Succeed())
+
+			_, err := reconciler.Reconcile(ctx, req)
+			Expect(err).NotTo(HaveOccurred())
+
+			// Check finalizer is removed
+			err = k8sclient.Get(ctx, req.NamespacedName, cluster)
+			Expect(err).To(HaveOccurred())
+			Expect(errors.IsNotFound(err)).To(BeTrue(), "cluster should be gone as finalizer is removed")
+		})
+	})
+})
diff --git a/internal/cmd/controller/reconciler/suite_test.go b/internal/cmd/controller/reconciler/suite_test.go
index e454ae92..b7b5f8bd 100644
--- a/internal/cmd/controller/reconciler/suite_test.go
+++ b/internal/cmd/controller/reconciler/suite_test.go
@@ -7,7 +7,7 @@ import (
 	. "github.com/onsi/gomega"
 )
 
-func TestScheduler(t *testing.T) {
+func TestControllers(t *testing.T) {
 	RegisterFailHandler(Fail)
-	RunSpecs(t, "Fleet Scheduler Controller Suite")
+	RunSpecs(t, "Fleet Controllers Suite")
 }