Repository: karmada-io/karmada

Issue:
For the scheduler APIEnablement plugin, scheduled resources are never deleted under any situation
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:

For the scheduler `APIEnablement` plugin, scheduled resources are never deleted under any situation.

**Why is this needed**:

The current logic of the `APIEnablement` plugin is that when the cluster API list is trusted, if the GVK (Group, Version, Kind) of a resource is not in the API list, the cluster will be filtered out. If the previous scheduling result included this cluster, the target resource in the cluster will be deleted.

This handling logic may not be suitable in the following scenario: A user maintains a CRD (Custom Resource Definition) resource in a member cluster and then propagates the CR (Custom Resource) of that CRD through the Karmada control plane. If the user accidentally deletes the CRD resource in the cluster and triggers the scheduler to reschedule the CR resource, even if the user reinstall the CRD resource in the member cluster, the CR resource will not be able to be recreated in the member cluster.

Therefore, to make the user's resources more stable, we can modify the logic of the `APIEnablement` plugin so that it never deletes scheduled resources under any circumstances.




Discussion:
@RainbowMango: This proposal attempts to refine PR #5216 to make the [APIEnablement plugin](https://github.com/karmada-io/karmada/blob/afc4b4fc226aa9b24f18b2ead1ad594c8629ee0b/pkg/scheduler/framework/plugins/apienablement/api_enablement.go#L62-L68) more predictable and avoid unexpected removal of workloads.

@yanfeng1992 @whitewindmills any thoughts?@XiShanYongYe-Chang: In favor of #6462 
/assign @abhinav-1305 @karmada-bot: @XiShanYongYe-Chang: GitHub didn't allow me to assign the following users: abhinav-1305.

Note that only [karmada-io members](https://github.com/orgs/karmada-io/people) with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see [the contributor guide](https://git.k8s.io/community/contributors/guide/first-contribution.md#issue-assignment-in-github)

<details>

In response to [this](https://github.com/karmada-io/karmada/issues/6461#issuecomment-2979313392):

>In favor of #6462 
>/assign @abhinav-1305 


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@whitewindmills: +1
we should rule out the potential risks of APIENABLEMENT. as for the legacy APIENABLEMENT status design, it can be used as an operation and maintenance troubleshooting measure for administrators.

Solution patch:
diff --git a/pkg/scheduler/framework/plugins/apienablement/api_enablement.go b/pkg/scheduler/framework/plugins/apienablement/api_enablement.go
index 791180f63..6e6a4ba65 100644
--- a/pkg/scheduler/framework/plugins/apienablement/api_enablement.go
+++ b/pkg/scheduler/framework/plugins/apienablement/api_enablement.go
@@ -19,7 +19,6 @@ package apienablement
 import (
 	"context"
 
-	"k8s.io/apimachinery/pkg/api/meta"
 	"k8s.io/klog/v2"
 
 	clusterv1alpha1 "github.com/karmada-io/karmada/pkg/apis/cluster/v1alpha1"
@@ -55,15 +54,21 @@ func (p *APIEnablement) Filter(
 	_ *workv1alpha2.ResourceBindingStatus,
 	cluster *clusterv1alpha1.Cluster,
 ) *framework.Result {
-	if helper.IsAPIEnabled(cluster.Status.APIEnablements, bindingSpec.Resource.APIVersion, bindingSpec.Resource.Kind) {
+	// If the cluster is already in the target list, always allow it to pass
+	// This ensures the scheduler never deletes scheduled resources by this plugin.
+	// In case of the required APIs(CRDs) are accidentally deleted from member cluster, Karmada
+	// controllers will try recreating these resources once the API becomes available again.
+	if bindingSpec.TargetContains(cluster.Name) {
 		return framework.NewResult(framework.Success)
 	}
 
-	// Let the cluster pass if it is already on the list of schedule result and the cluster's
-	// API enablements is incomplete, to avoid the issue that cluster be accidentally removed
-	// due to untrusted API enablements.
-	if bindingSpec.TargetContains(cluster.Name) &&
-		!meta.IsStatusConditionTrue(cluster.Status.Conditions, clusterv1alpha1.ClusterConditionCompleteAPIEnablements) {
+	// For new scheduling decisions, check if the API is enabled to ensure the application
+	// is deployed to a cluster that supports all required APIs.
+	// Note: Although controllers may not always get the complete API list,
+	// we enforce strict checks to maintain consistency. This may occasionally
+	// exclude clusters prematurely. Users requiring a specific number of target
+	// clusters should use SpreadConstraints(in PropagationPolicy) to meet their requirements.
+	if helper.IsAPIEnabled(cluster.Status.APIEnablements, bindingSpec.Resource.APIVersion, bindingSpec.Resource.Kind) {
 		return framework.NewResult(framework.Success)
 	}
 



Test patch:
diff --git a/pkg/scheduler/framework/plugins/apienablement/api_enablement_test.go b/pkg/scheduler/framework/plugins/apienablement/api_enablement_test.go
index cf68ae0c0..52e171bcc 100644
--- a/pkg/scheduler/framework/plugins/apienablement/api_enablement_test.go
+++ b/pkg/scheduler/framework/plugins/apienablement/api_enablement_test.go
@@ -93,7 +93,7 @@ func TestAPIEnablement_Filter(t *testing.T) {
 			expectError:  true,
 		},
 		{
-			name: "cluster in target list with incomplete API enablements",
+			name: "cluster in target list with API not enabled",
 			bindingSpec: &workv1alpha2.ResourceBindingSpec{
 				Resource: workv1alpha2.ObjectReference{
 					APIVersion: "custom.io/v1",
@@ -110,10 +110,14 @@ func TestAPIEnablement_Filter(t *testing.T) {
 					Name: "cluster1",
 				},
 				Status: clusterv1alpha1.ClusterStatus{
-					Conditions: []metav1.Condition{
+					APIEnablements: []clusterv1alpha1.APIEnablement{
 						{
-							Type:   clusterv1alpha1.ClusterConditionCompleteAPIEnablements,
-							Status: metav1.ConditionFalse,
+							GroupVersion: "apps/v1",
+							Resources: []clusterv1alpha1.APIResource{
+								{
+									Kind: "Deployment",
+								},
+							},
 						},
 					},
 				},