Repository: apecloud/kubeblocks

Issue:
[BUG]delete restored mongodb cluster failed
**Describe the bug**
```
Kubernetes: v1.31.1-aliyun.1
KubeBlocks: 1.0.0-beta.48
kbcli: 1.0.0-beta.21
```

**To Reproduce**
Steps to reproduce the behavior:
1. Create mongodb cluster
```
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: mongodb-lkpaky
  namespace: default
spec:
  clusterDef: mongodb
  topology: replicaset
  terminationPolicy: DoNotTerminate
  componentSpecs:
    - name: mongodb
      serviceVersion: 8.0.4
      replicas: 3
      resources:
        limits:
          cpu: 100m
          memory: 0.5Gi
        requests:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
```
2. backup&restore
```
kbcli cluster backup mongodb-lkpaky --method datafile --namespace default
Backup backup-default-mongodb-lkpaky-20250519111839 created successfully, you can view the progress:
	kbcli cluster list-backups --name=backup-default-mongodb-lkpaky-20250519111839 -n default

kbcli cluster restore mongodb-lkpaky-backup --backup backup-default-mongodb-lkpaky-20250519111839 --namespace default
Cluster mongodb-lkpaky-backup created

k get cluster -A
NAMESPACE   NAME                    CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
default     mongodb-lkpaky          mongodb              DoNotTerminate       Running   15m
default     mongodb-lkpaky-backup   mongodb              DoNotTerminate       Running   10m
```

3. delete restored cluster
```
kubectl patch cluster mongodb-lkpaky-backup -p '{"spec":{"terminationPolicy":"WipeOut"}}' --type="merge"
cluster.apps.kubeblocks.io/mongodb-lkpaky-backup patched
tianyue@192 kubeblocks-addons % k delete cluster mongodb-lkpaky-backup

k get cluster -A
NAMESPACE   NAME                    CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
default     mongodb-lkpaky          mongodb              DoNotTerminate       Running    20m
default     mongodb-lkpaky-backup   mongodb              WipeOut              Deleting   15m
```

4. See error
```
2025-05-19T03:33:14.317Z	ERROR	delete *v1.InstanceSet error: mongodb-lkpaky-backup-mongodb	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet", "InstanceSet": {"name":"mongodb-lkpaky-backup-mongodb","namespace":"default"}, "namespace": "default", "name": "mongodb-lkpaky-backup-mongodb", "reconcileID": "cc298cbd-08c6-43a8-b3a5-d3577d04ff7d", "InstanceSet": {"name":"mongodb-lkpaky-backup-mongodb","namespace":"default"}, "error": "Operation cannot be fulfilled on instancesets.workloads.kubeblocks.io \"mongodb-lkpaky-backup-mongodb\": StorageError: invalid object, Code: 4, Key: /cddd4aa2b2316413ea27eba609d4d1e6e/workloads.kubeblocks.io/instancesets/default/mongodb-lkpaky-backup-mongodb, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 775655c6-9d5b-402e-bcaa-01db244dc1ad, UID in object meta: "}github.com/apecloud/kubeblocks/pkg/controller/kubebuilderx.(*PlanBuilder).deleteObject
	/src/pkg/controller/kubebuilderx/plan_builder.go:292
github.com/apecloud/kubeblocks/pkg/controller/kubebuilderx.(*PlanBuilder).defaultWalkFunc
	/src/pkg/controller/kubebuilderx/plan_builder.go:252
github.com/apecloud/kubeblocks/pkg/controller/kubebuilderx.(*Plan).Execute
	/src/pkg/controller/kubebuilderx/plan_builder.go:226
github.com/apecloud/kubeblocks/pkg/controller/kubebuilderx.(*controller).Commit
	/src/pkg/controller/kubebuilderx/controller.go:123
github.com/apecloud/kubeblocks/controllers/workloads.(*InstanceSetReconciler).Reconcile
	/src/controllers/workloads/instanceset_controller.go:92
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2025-05-19T03:33:14.323Z	INFO	deleting the sub-resource kinds: [Service ]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongodb-lkpaky-backup","namespace":"default"}, "namespace": "default", "name": "mongodb-lkpaky-backup", "reconcileID": "7c69b434-614c-4e98-9eb6-6803d6490610", "cluster": {"name":"mongodb-lkpaky-backup","namespace":"default"}}
```

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.


Hints from issue comments:
@wangyelei: component is not deleting but kb reports

![Image](https://github.com/user-attachments/assets/eb3ff4fa-d382-4ea7-9172-eefac4c2f12c)@wangyelei: missing mongodb component@wangyelei: a workroud: delete the resources which list by `kubectl get pvc,cm,secret,svc,pod | grep mongodb-lkpaky-backup ` manually

Solution patch:
diff --git a/controllers/apps/cluster/cluster_controller.go b/controllers/apps/cluster/cluster_controller.go
index e048b831a..86a9c7e5e 100644
--- a/controllers/apps/cluster/cluster_controller.go
+++ b/controllers/apps/cluster/cluster_controller.go
@@ -126,6 +126,7 @@ func (r *ClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 	// TODO: transformers are vertices, theirs' dependencies are edges, make plan Build stage a DAG.
 	plan, errBuild := planBuilder.
 		AddTransformer(
+			&clusterTerminationPolicyTransformer{},
 			// handle cluster deletion
 			&clusterDeletionTransformer{},
 			// update finalizer and definition labels
diff --git a/controllers/apps/cluster/transformer_cluster_deletion.go b/controllers/apps/cluster/transformer_cluster_deletion.go
index 297b3e8bf..269320cfa 100644
--- a/controllers/apps/cluster/transformer_cluster_deletion.go
+++ b/controllers/apps/cluster/transformer_cluster_deletion.go
@@ -80,7 +80,7 @@ func (t *clusterDeletionTransformer) Transform(ctx graph.TransformContext, dag *
 	if len(deleteSet) > 0 {
 		// wait for the components to be deleted to trigger the next reconcile
 		transCtx.Logger.Info(fmt.Sprintf("wait for the components and shardings to be deleted: %v", deleteSet))
-		return nil
+		return graph.ErrPrematureStop
 	}
 
 	// then list all the others objects owned by this cluster in cache, and delete them all
diff --git a/controllers/apps/cluster/transformer_cluster_termination_policy.go b/controllers/apps/cluster/transformer_cluster_termination_policy.go
new file mode 100644
index 000000000..0b28490d4
--- /dev/null
+++ b/controllers/apps/cluster/transformer_cluster_termination_policy.go
@@ -0,0 +1,63 @@
+/*
+Copyright (C) 2022-2025 ApeCloud Co., Ltd
+
+This file is part of KubeBlocks project
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package cluster
+
+import (
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	appsv1 "github.com/apecloud/kubeblocks/apis/apps/v1"
+	"github.com/apecloud/kubeblocks/pkg/constant"
+	"github.com/apecloud/kubeblocks/pkg/controller/graph"
+	"github.com/apecloud/kubeblocks/pkg/controller/model"
+)
+
+type clusterTerminationPolicyTransformer struct{}
+
+var _ graph.Transformer = &clusterTerminationPolicyTransformer{}
+
+func (t *clusterTerminationPolicyTransformer) Transform(ctx graph.TransformContext, dag *graph.DAG) error {
+	transCtx, _ := ctx.(*clusterTransformContext)
+	cluster := transCtx.OrigCluster
+	if !cluster.IsDeleting() {
+		return nil
+	}
+
+	compList := &appsv1.ComponentList{}
+	ml := client.MatchingLabels{constant.AppInstanceLabelKey: cluster.Name}
+	if err := transCtx.Client.List(transCtx.Context, compList, client.InNamespace(cluster.Namespace), ml); err != nil {
+		return err
+	}
+
+	hasUpdate := false
+	graphCli, _ := transCtx.Client.(model.GraphClient)
+	for i, comp := range compList.Items {
+		if cluster.Spec.TerminationPolicy != comp.Spec.TerminationPolicy {
+			hasUpdate = true
+			obj := compList.Items[i].DeepCopy()
+			obj.Spec.TerminationPolicy = cluster.Spec.TerminationPolicy
+			graphCli.Update(dag, &compList.Items[i], obj)
+		}
+	}
+
+	if hasUpdate {
+		return graph.ErrPrematureStop
+	}
+	return nil
+}
diff --git a/controllers/apps/component/transformer_component_deletion.go b/controllers/apps/component/transformer_component_deletion.go
index fd64bf3d5..a2129b59f 100644
--- a/controllers/apps/component/transformer_component_deletion.go
+++ b/controllers/apps/component/transformer_component_deletion.go
@@ -113,7 +113,7 @@ func (t *componentDeletionTransformer) deleteCompResources(transCtx *componentTr
 		}
 		// wait for the workloads to be deleted to trigger the next reconcile
 		transCtx.Logger.Info(fmt.Sprintf("wait for the workloads to be deleted: %v", workloads))
-		return nil
+		return graph.ErrPrematureStop
 	}
 
 	// secondly, delete the other sub-resources owned by the component



Test patch:
diff --git a/controllers/apps/cluster/transformer_cluster_deletion_test.go b/controllers/apps/cluster/transformer_cluster_deletion_test.go
index c8482e2ac..23cc85881 100644
--- a/controllers/apps/cluster/transformer_cluster_deletion_test.go
+++ b/controllers/apps/cluster/transformer_cluster_deletion_test.go
@@ -20,6 +20,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package cluster
 
 import (
+	"errors"
 	"slices"
 	"time"
 
@@ -121,7 +122,8 @@ var _ = Describe("clusterDeletionTransformer", func() {
 
 		transformer := &clusterDeletionTransformer{}
 		err := transformer.Transform(transCtx, dag)
-		Expect(err).Should(BeNil())
+		Expect(err).ShouldNot(BeNil())
+		Expect(errors.Is(err, graph.ErrPrematureStop)).Should(BeTrue())
 		Expect(dag.Vertices()).Should(HaveLen(1 + 3))
 	})
 
@@ -149,7 +151,8 @@ var _ = Describe("clusterDeletionTransformer", func() {
 		})
 		dag = newDag(transCtx.Client.(model.GraphClient))
 		err = transformer.Transform(transCtx, dag)
-		Expect(err).Should(BeNil())
+		Expect(err).ShouldNot(BeNil())
+		Expect(errors.Is(err, graph.ErrPrematureStop)).Should(BeTrue())
 		Expect(dag.Vertices()).Should(HaveLen(1 + 1))
 
 		// delete component 3