Repository: kubernetes/kubernetes (language: go)

Issue:
integration tests are painfully slow
This job is often taking well over an hour even with 8 cores and 20Gi assigned in prow. It used to be more like 30m or less.

(because these tests cases are mostly kubectl and apiserver)

A handful of specific tests seem to be large outliers

the CEL tests alone take 11+ minutes
> k8s.io/kubernetes/test/integration/apiserver: cel | 11m13s

/sig api-machinery

cc @jpbetz @cici37 

and OIDC
> k8s.io/kubernetes/test/integration/apiserver: oidc | 11m16s

/sig auth


> k8s.io/kubernetes/test/integration/controlplane: transformation | 14m33s

> k8s.io/kubernetes/test/integration: examples | 7m47s


api machinery again?


> k8s.io/kubernetes/test/integration: job | 14m16s

/sig apps

> k8s.io/kubernetes/test/integration: scheduler_perf | 9m34s

/sig scheduling

Most of the SIG CLI tests take a few seconds

NOTE: I only skimmed a few of the excessive results here, you can see all of the timings if you expand the "174/174 tests passed!" field

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/126191/pull-kubernetes-integration/1813929954224640000






can we do something about this folks?

/kind bug


Discussion / hints:
This sort of thing hurts the contributor experience, we typically would not permit a presubmit that took longer than 1 hour because that creates an excessively long feedback loop. (We aim for <= 30m, e2e-gce has typically been the outlier at 45m+)

If you have really slow expensive tests and they must stay that way, they should probably be broken out into another job and we should strongly consider if they need to run on every single PR push versus on merged code. It is not cheap to run this.

I would really appreciate folks taking a look at cutting down these test times.
> k8s.io/kubernetes/test/integration/controlplane: transformation

cc @enj
> k8s.io/kubernetes/test/integration: scheduler_perf

cc @alculquicondor 
> k8s.io/kubernetes/test/integration: job

cc @alculquicondor @mimowo
> k8s.io/kubernetes/test/integration/apiserver: oidc

cc @enj @aramase
we need to start splitting between periodics and presubmit, we can tag slows to run only on periodics
A relatively simple optimization for job integration tests might be to reuse the API server across test cases within a test.
> It used to be more like 30m or less 

Not for a while. I remember that when I started working on running integration tests with race detection enabled (https://github.com/kubernetes/kubernetes/pull/116980) more than a year ago it was already close to an hour.

> k8s.io/kubernetes/test/integration: scheduler_perf | 9m34s

I was involved with  `scheduler_perf` running as integration test. I added that because it provided some valuable additional coverage of more complex scheduling scenarios. The runtime can be reduced by picking different configurations in performance_config.yaml. What would be an acceptable maximum duration?

> we need to start splitting between periodics and presubmit, we can tag slows to run only on periodics

How? I think we currently have no way to tell `make test-integration` what it should and shouldn't run - besides `WHAT`, but that would be a pain to maintain in a job definition.

> A relatively simple optimization for job integration tests might be to reuse the API server across test cases within a test.

That's already done.
> How? I think we currently have no way to tell `make test-integration` what it should and shouldn't run - besides `WHAT`, but that would be a pain to maintain in a job definition.
> 
> >

we did at one point, and *should* still

in the test:
```
		if testing.Short() {
			t.Skip("skipping expensive test")
		}
```

in test-integration:
`KUBE_TEST_ARGS="${SHORT:--short=true}...`

in test-infra for post-submit / periodic:
```
      env:
      - name: SHORT
        value: --short=false
```
> Not for a while. I remember that when I started working on running integration tests with race detection enabled (https://github.com/kubernetes/kubernetes/pull/116980) more than a year ago it was already close to an hour.

It was for many many years though, and we should fix that. I'm just looking through the results now and flagging that we've let it regress, the fact that the regression didn't happen overnight doesn't mean we shouldn't fix it.

We can't just keep regressing presubmits longer and longer. e2e-gce is also too slow and should be resolved.

If we can't move the tests out, then we need to make them faster or take more advantage of available CI hardware / parallelism.


> How? I think we currently have no way to tell make test-integration what it should and shouldn't run - besides WHAT, but that would be a pain to maintain in a job definition.

WHAT is an option but shouldn't be necessary due to go's built-in equivalent to [Slow], see Jordan's comment.


> I was involved with scheduler_perf running as integration test. I added that because it provided some valuable additional coverage of more complex scheduling scenarios. The runtime can be reduced by picking different configurations in performance_config.yaml. What would be an acceptable maximum duration?

Ideally a few minutes or less, preferably faster.

Also pointing out: this job is running on the EKS cluster with a fraction of the cores allocated (8/16?) since some of the other changes made, versus the sole tenant on the GKE cluster previously (7/8 cores assigned), so performance tests are somewhat less meaningful due to noisy neighbor issues.


> It was for many many years though, and we should fix that.

I don't disagree. I just thought that "below 1 hour" was the guideline and it was hitting that, at least when I looked.

> `if testing.Short() {`

Neat, I didn't know that one. We don't have any periodic integration test job, do we we?

> performance tests are somewhat less meaningful due to noisy neighbor issues.

scheduler_perf runs in "benchmarking" mode in some other job. During integration testing it runs different configurations of the same scenarios as unit tests.

> We don't have any periodic integration test job, do we we?

We do: ci-kubernetes-integration-master
> I was involved with `scheduler_perf` running as integration test. I added that because it provided some valuable additional coverage of more complex scheduling scenarios. The runtime can be reduced by picking different configurations in performance_config.yaml. What would be an acceptable maximum duration?

Yeah, we could mark less tests as "fast" so that they don't run as part of the integration job. Should we cap it at 5m? 2m?
> > A relatively simple optimization for job integration tests might be to reuse the API server across test cases within a test.
> 
> That's already done.

Is it? In some tests (see [example](https://github.com/kubernetes/kubernetes/blob/dda657b5982e8f9102b8df5931344262b0793163/test/integration/job/job_test.go#L431-L432)) we call `setup` inside `t.Run` which IIUC spawns the API server inside, see [here](https://github.com/kubernetes/kubernetes/blob/dda657b5982e8f9102b8df5931344262b0793163/test/integration/job/job_test.go#L4232). Though, I'm not sure what is the actual benefit, probably will not change the overall picture much.
> Yeah, we could mark less tests as "fast" so that they don't run as part of the integration job. Should we cap it at 5m? 2m?

Let's start at 5m and go from there.
/assign @macsko
(for the scheduler_perf side)
> >    That's already done.
>
> Is it?

Perhaps not everywhere, but definitely in several tests.

scheduler_perf pov:

We have two ways of running the tests:
- benchmark - creating apiserver, etcd and other components per each test case separately ([ref](https://github.com/kubernetes/kubernetes/blob/01eb9f47541156a38e4b28759977c06f07f08012/test/integration/scheduler_perf/scheduler_perf.go#L835-L846))
- integration - creating etcd **once** at the beginning of the tests ([ref](https://github.com/kubernetes/kubernetes/blob/01eb9f47541156a38e4b28759977c06f07f08012/test/integration/scheduler_perf/scheduler_test.go#L42)) and create apiserver for each unique config ([ref](https://github.com/kubernetes/kubernetes/blob/01eb9f47541156a38e4b28759977c06f07f08012/test/integration/scheduler_perf/scheduler_test.go#L80))

I found that running the benchmark way as integration tests with label set to `integration-test`, so running the same test cases, reduces the testing time from 530s to 230s in my case. 

I think the difference might come from cleaning up etcd after each test case which can be time consuming ([ref](https://github.com/kubernetes/kubernetes/blob/01eb9f47541156a38e4b28759977c06f07f08012/test/integration/scheduler_perf/scheduler_perf.go#L1031-L1034)). 

We could unify the tests structure to benchmark way.
@macsko: this issue isn't about the "benchmark" mode. That runs in a different job. Restarting etcd and the apiserver during benchmarking is intentional because it reduces the risk that one scenario influences another.

> We could unify the tests structure to benchmark way.

That would make it run slower - the opposite of what we want.

> I found that running the benchmark way as integration tests with label set to integration-test, so running the same test cases, reduces the testing time from 530s to 230s in my case.

@pohly It looks like, experimentally, just recreating the etcd is faster than removing the objects from the one etcd instance.
Ah, okay. That could be true because these tests have a lot of objects to delete, which is done one-by-one instead using deletecollection.

Fix patch:
diff --git a/staging/src/k8s.io/apiserver/pkg/admission/plugin/policy/generic/policy_source.go b/staging/src/k8s.io/apiserver/pkg/admission/plugin/policy/generic/policy_source.go
index 9b2e2146a8f0e..ca6cdc884fc77 100644
--- a/staging/src/k8s.io/apiserver/pkg/admission/plugin/policy/generic/policy_source.go
+++ b/staging/src/k8s.io/apiserver/pkg/admission/plugin/policy/generic/policy_source.go
@@ -41,6 +41,13 @@ import (
 	"k8s.io/klog/v2"
 )
 
+// Interval for refreshing policies.
+// TODO: Consider reducing this to a shorter duration or replacing this entirely
+// with checks that detect when a policy change took effect.
+const policyRefreshIntervalDefault = 1 * time.Second
+
+var policyRefreshInterval = policyRefreshIntervalDefault
+
 type policySource[P runtime.Object, B runtime.Object, E Evaluator] struct {
 	ctx                context.Context
 	policyInformer     generic.Informer[P]
@@ -122,6 +129,15 @@ func NewPolicySource[P runtime.Object, B runtime.Object, E Evaluator](
 	return res
 }
 
+// SetPolicyRefreshIntervalForTests allows the refresh interval to be overridden during tests.
+// This should only be called from tests.
+func SetPolicyRefreshIntervalForTests(interval time.Duration) func() {
+	policyRefreshInterval = interval
+	return func() {
+		policyRefreshInterval = policyRefreshIntervalDefault
+	}
+}
+
 func (s *policySource[P, B, E]) Run(ctx context.Context) error {
 	if s.ctx != nil {
 		return fmt.Errorf("policy source already running")
@@ -178,7 +194,7 @@ func (s *policySource[P, B, E]) Run(ctx context.Context) error {
 	// and needs to be recompiled
 	go func() {
 		// Loop every 1 second until context is cancelled, refreshing policies
-		wait.Until(s.refreshPolicies, 1*time.Second, ctx.Done())
+		wait.Until(s.refreshPolicies, policyRefreshInterval, ctx.Done())
 	}()
 
 	<-ctx.Done()


Test patch:
diff --git a/test/integration/apiserver/cel/validatingadmissionpolicy_test.go b/test/integration/apiserver/cel/validatingadmissionpolicy_test.go
index c27cc1754feda..2dd810dd9daa9 100644
--- a/test/integration/apiserver/cel/validatingadmissionpolicy_test.go
+++ b/test/integration/apiserver/cel/validatingadmissionpolicy_test.go
@@ -57,6 +57,7 @@ import (
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/sets"
 	"k8s.io/apimachinery/pkg/util/wait"
+	"k8s.io/apiserver/pkg/admission/plugin/policy/generic"
 	"k8s.io/client-go/dynamic"
 	clientset "k8s.io/client-go/kubernetes"
 
@@ -66,17 +67,34 @@ import (
 	rbacv1 "k8s.io/api/rbac/v1"
 )
 
-// Test_ValidateNamespace_NoParams tests a ValidatingAdmissionPolicy that validates creation of a Namespace with no params.
-func Test_ValidateNamespace_NoParams(t *testing.T) {
-	forbiddenReason := metav1.StatusReasonForbidden
+// Short term fix to refresh the policy source cache faster for tests
+// until we reduce the polling interval by default.
+const policyRefreshInterval = 10 * time.Millisecond
+
+// Test_ValidateNamespace_NoParams_Success tests a ValidatingAdmissionPolicy that validates creation of a Namespace
+// with no params via happy path.
+func Test_ValidateNamespace_NoParams_Success(t *testing.T) {
+	resetPolicyRefreshInterval := generic.SetPolicyRefreshIntervalForTests(policyRefreshInterval)
+	defer resetPolicyRefreshInterval()
+	featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
+	server, err := apiservertesting.StartTestServer(t, nil, []string{
+		"--enable-admission-plugins", "ValidatingAdmissionPolicy",
+	}, framework.SharedEtcd())
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer server.TearDownFn()
+
+	config := server.ClientConfig
+	client, err := clientset.NewForConfig(config)
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	testcases := []struct {
 		name          string
 		policy        *admissionregistrationv1.ValidatingAdmissionPolicy
 		policyBinding *admissionregistrationv1.ValidatingAdmissionPolicyBinding
-		namespace     *v1.Namespace
-		err           string
-		failureReason metav1.StatusReason
 	}{
 		{
 			name: "namespace name contains suffix enforced by validating admission policy, using object metadata fields",
@@ -86,160 +104,171 @@ func Test_ValidateNamespace_NoParams(t *testing.T) {
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err: "",
 		},
 		{
-			name: "namespace name does NOT contain suffix enforced by validating admission policyusing, object metadata fields",
+			name: "namespace name contains suffix enforced by validating admission policy, using request field",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "object.metadata.name.endsWith('k8s')",
+					Expression: "request.name.endsWith('k8s')",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-foobar",
-				},
-			},
-			err:           "namespaces \"test-foobar\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: failed expression: object.metadata.name.endsWith('k8s')",
-			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
-			name: "namespace name does NOT contain suffix enforced by validating admission policy using object metadata fields, AND validating expression returns StatusReasonForbidden",
+			name: "namespace name does NOT contains suffix enforced by validating admission policy, using request field",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "object.metadata.name.endsWith('k8s')",
-					Reason:     &forbiddenReason,
+					Expression: "request.name.endsWith('k8s')",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "forbidden-test-foobar",
-				},
-			},
-			err:           "namespaces \"forbidden-test-foobar\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: failed expression: object.metadata.name.endsWith('k8s')",
-			failureReason: metav1.StatusReasonForbidden,
 		},
 		{
-			name: "namespace name contains suffix enforced by validating admission policy, using request field",
+			name: "runtime error when validating namespace, but failurePolicy=Ignore",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "request.name.endsWith('k8s')",
+					Expression: "object.nonExistentProperty == 'someval'",
 				},
-			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
+			}, withFailurePolicy(admissionregistrationv1.Ignore, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err: "",
 		},
 		{
-			name: "namespace name does NOT contains suffix enforced by validating admission policy, using request field",
+			name: "with check against unguarded params using has() and default check",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "request.name.endsWith('k8s')",
+					Expression: "(has(params.metadata) && has(params.metadata.name) && object.metadata.name.startsWith(params.metadata.name)) || object.metadata.name.endsWith('k8s')",
 				},
-			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
+			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err: "",
 		},
 		{
-			name: "runtime error when validating namespace, but failurePolicy=Ignore",
+			name: "with check against null params and default check",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "object.nonExistentProperty == 'someval'",
+					Expression: "(params != null && object.metadata.name.startsWith(params.metadata.name)) || object.metadata.name.endsWith('k8s')",
 				},
-			}, withFailurePolicy(admissionregistrationv1.Ignore, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
+			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err: "",
 		},
 		{
-			name: "runtime error when validating namespace, but failurePolicy=Fail",
+			name: "with check against namespaceObject",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "object.nonExistentProperty == 'someval'",
+					Expression: "namespaceObject == null", // because namespace itself is cluster-scoped.
 				},
-			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
+			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
+		},
+	}
+	for i, testcase := range testcases {
+		t.Run(testcase.name, func(t *testing.T) {
+			policy := withWaitReadyConstraintAndExpression(testcase.policy)
+			if _, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}); err != nil {
+				t.Fatal(err)
+			}
+			if err := createAndWaitReady(t, client, testcase.policyBinding, nil); err != nil {
+				t.Fatal(err)
+			}
+
+			nsName := fmt.Sprintf("test-%d-k8s", i)
+			ns := &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
+					Name: nsName,
 				},
-			},
-			err:           "namespaces \"test-k8s\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: expression 'object.nonExistentProperty == 'someval'' resulted in error: no such key: nonExistentProperty",
-			failureReason: metav1.StatusReasonInvalid,
-		},
+			}
+			_, err = client.CoreV1().Namespaces().Create(context.TODO(), ns, metav1.CreateOptions{})
+
+			checkExpectedError(t, err, "")
+			checkFailureReason(t, err, "")
+			if err := cleanupPolicy(t, client, policy, testcase.policyBinding); err != nil {
+				t.Errorf("error while cleaning up policy and its bindings: %v", err)
+			}
+		})
+	}
+}
+
+// Test_ValidateNamespace_NoParams_Failures tests a ValidatingAdmissionPolicy that fails creation of a Namespace with no params.
+func Test_ValidateNamespace_NoParams_Failures(t *testing.T) {
+	resetPolicyRefreshInterval := generic.SetPolicyRefreshIntervalForTests(policyRefreshInterval)
+	defer resetPolicyRefreshInterval()
+	featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
+	server, err := apiservertesting.StartTestServer(t, nil, []string{
+		"--enable-admission-plugins", "ValidatingAdmissionPolicy",
+	}, framework.SharedEtcd())
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer server.TearDownFn()
+
+	config := server.ClientConfig
+	client, err := clientset.NewForConfig(config)
+	if err != nil {
+		t.Fatal(err)
+	}
+	forbiddenReason := metav1.StatusReasonForbidden
+
+	testcases := []struct {
+		name          string
+		policy        *admissionregistrationv1.ValidatingAdmissionPolicy
+		policyBinding *admissionregistrationv1.ValidatingAdmissionPolicyBinding
+		namespace     *v1.Namespace
+		err           string
+		failureReason metav1.StatusReason
+	}{
 		{
-			name: "runtime error due to unguarded params",
+			name: "namespace name does NOT contain suffix enforced by validating admission policyusing, object metadata fields",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "object.metadata.name.startsWith(params.metadata.name)",
+					Expression: "object.metadata.name.endsWith('k8s')",
 				},
-			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
+			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
 			namespace: &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
+					Name: "test-foobar",
 				},
 			},
-			err:           "namespaces \"test-k8s\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: expression 'object.metadata.name.startsWith(params.metadata.name)' resulted in error: no such key: metadata",
+			err:           "namespaces \"test-foobar\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: failed expression: object.metadata.name.endsWith('k8s')",
 			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
-			name: "with check against unguarded params using has()",
+			name: "namespace name does NOT contain suffix enforced by validating admission policy using object metadata fields, AND validating expression returns StatusReasonForbidden",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "has(params.metadata) && has(params.metadata.name) && object.metadata.name.endsWith(params.metadata.name)",
+					Expression: "object.metadata.name.endsWith('k8s')",
+					Reason:     &forbiddenReason,
 				},
-			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
+			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
 			namespace: &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
+					Name: "forbidden-test-foobar",
 				},
 			},
-			err:           "namespaces \"test-k8s\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: failed expression: has(params.metadata) && has(params.metadata.name) && object.metadata.name.endsWith(params.metadata.name)",
-			failureReason: metav1.StatusReasonInvalid,
+			err:           "namespaces \"forbidden-test-foobar\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: failed expression: object.metadata.name.endsWith('k8s')",
+			failureReason: metav1.StatusReasonForbidden,
 		},
 		{
-			name: "with check against null params",
+			name: "runtime error when validating namespace, but failurePolicy=Fail",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "(params != null && object.metadata.name.endsWith(params.metadata.name))",
+					Expression: "object.nonExistentProperty == 'someval'",
 				},
-			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
+			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
 			namespace: &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
 					Name: "test-k8s",
 				},
 			},
-			err:           "namespaces \"test-k8s\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: failed expression: (params != null && object.metadata.name.endsWith(params.metadata.name))",
+			err:           "namespaces \"test-k8s\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: expression 'object.nonExistentProperty == 'someval'' resulted in error: no such key: nonExistentProperty",
 			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
-			name: "with check against unguarded params using has() and default check",
+			name: "runtime error due to unguarded params",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "(has(params.metadata) && has(params.metadata.name) && object.metadata.name.startsWith(params.metadata.name)) || object.metadata.name.endsWith('k8s')",
+					Expression: "object.metadata.name.startsWith(params.metadata.name)",
 				},
 			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
@@ -248,13 +277,14 @@ func Test_ValidateNamespace_NoParams(t *testing.T) {
 					Name: "test-k8s",
 				},
 			},
-			err: "",
+			err:           "namespaces \"test-k8s\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: expression 'object.metadata.name.startsWith(params.metadata.name)' resulted in error: no such key: metadata",
+			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
-			name: "with check against null params and default check",
+			name: "with check against unguarded params using has()",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "(params != null && object.metadata.name.startsWith(params.metadata.name)) || object.metadata.name.endsWith('k8s')",
+					Expression: "has(params.metadata) && has(params.metadata.name) && object.metadata.name.endsWith(params.metadata.name)",
 				},
 			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
@@ -263,13 +293,14 @@ func Test_ValidateNamespace_NoParams(t *testing.T) {
 					Name: "test-k8s",
 				},
 			},
-			err: "",
+			err:           "namespaces \"test-k8s\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: failed expression: has(params.metadata) && has(params.metadata.name) && object.metadata.name.endsWith(params.metadata.name)",
+			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
-			name: "with check against namespaceObject",
+			name: "with check against null params",
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
-					Expression: "namespaceObject == null", // because namespace itself is cluster-scoped.
+					Expression: "(params != null && object.metadata.name.endsWith(params.metadata.name))",
 				},
 			}, withParams(configParamKind(), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
 			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
@@ -278,27 +309,12 @@ func Test_ValidateNamespace_NoParams(t *testing.T) {
 					Name: "test-k8s",
 				},
 			},
-			err: "",
+			err:           "namespaces \"test-k8s\" is forbidden: ValidatingAdmissionPolicy 'validate-namespace-suffix' with binding 'validate-namespace-suffix-binding' denied request: failed expression: (params != null && object.metadata.name.endsWith(params.metadata.name))",
+			failureReason: metav1.StatusReasonInvalid,
 		},
 	}
 	for _, testcase := range testcases {
 		t.Run(testcase.name, func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
-
-			server, err := apiservertesting.StartTestServer(t, nil, []string{
-				"--enable-admission-plugins", "ValidatingAdmissionPolicy",
-			}, framework.SharedEtcd())
-			if err != nil {
-				t.Fatal(err)
-			}
-			defer server.TearDownFn()
-
-			config := server.ClientConfig
-
-			client, err := clientset.NewForConfig(config)
-			if err != nil {
-				t.Fatal(err)
-			}
 			policy := withWaitReadyConstraintAndExpression(testcase.policy)
 			if _, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}); err != nil {
 				t.Fatal(err)
@@ -311,9 +327,13 @@ func Test_ValidateNamespace_NoParams(t *testing.T) {
 
 			checkExpectedError(t, err, testcase.err)
 			checkFailureReason(t, err, testcase.failureReason)
+			if err := cleanupPolicy(t, client, policy, testcase.policyBinding); err != nil {
+				t.Errorf("error while cleaning up policy and its bindings: %v", err)
+			}
 		})
 	}
 }
+
 func Test_ValidateAnnotationsAndWarnings(t *testing.T) {
 	testcases := []struct {
 		name             string
@@ -502,26 +522,53 @@ func Test_ValidateAnnotationsAndWarnings(t *testing.T) {
 // Test_ValidateNamespace_WithConfigMapParams tests a ValidatingAdmissionPolicy that validates creation of a Namespace,
 // using ConfigMap as a param reference.
 func Test_ValidateNamespace_WithConfigMapParams(t *testing.T) {
+	resetPolicyRefreshInterval := generic.SetPolicyRefreshIntervalForTests(policyRefreshInterval)
+	defer resetPolicyRefreshInterval()
+	featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
+	server, err := apiservertesting.StartTestServer(t, nil, []string{
+		"--enable-admission-plugins", "ValidatingAdmissionPolicy",
+	}, framework.SharedEtcd())
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer server.TearDownFn()
+
+	config := server.ClientConfig
+	client, err := clientset.NewForConfig(config)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	policyBinding := makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", "validate-namespace-suffix-param")
+	configMap := makeConfigParams("validate-namespace-suffix-param", map[string]string{
+		"namespaceSuffix": "k8s",
+	})
+	if _, err := client.CoreV1().ConfigMaps("default").Create(context.TODO(), configMap, metav1.CreateOptions{}); err != nil {
+		t.Fatal(err)
+	}
+
+	valPolicy := withValidations([]admissionregistrationv1.Validation{
+		{
+			Expression: "object.metadata.name.endsWith(params.data.namespaceSuffix)",
+		},
+	}, withFailurePolicy(admissionregistrationv1.Fail, withParams(configParamKind(), withNamespaceMatch(makePolicy("validate-namespace-suffix")))))
+	policy := withWaitReadyConstraintAndExpression(valPolicy)
+	if _, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}); err != nil {
+		t.Fatal(err)
+	}
+
+	if err := createAndWaitReady(t, client, policyBinding, nil); err != nil {
+		t.Fatal(err)
+	}
+
 	testcases := []struct {
 		name          string
-		policy        *admissionregistrationv1.ValidatingAdmissionPolicy
-		policyBinding *admissionregistrationv1.ValidatingAdmissionPolicyBinding
-		configMap     *v1.ConfigMap
 		namespace     *v1.Namespace
 		err           string
 		failureReason metav1.StatusReason
 	}{
 		{
 			name: "namespace name contains suffix enforced by validating admission policy",
-			policy: withValidations([]admissionregistrationv1.Validation{
-				{
-					Expression: "object.metadata.name.endsWith(params.data.namespaceSuffix)",
-				},
-			}, withFailurePolicy(admissionregistrationv1.Fail, withParams(configParamKind(), withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", "validate-namespace-suffix-param"),
-			configMap: makeConfigParams("validate-namespace-suffix-param", map[string]string{
-				"namespaceSuffix": "k8s",
-			}),
 			namespace: &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
 					Name: "test-k8s",
@@ -531,15 +578,6 @@ func Test_ValidateNamespace_WithConfigMapParams(t *testing.T) {
 		},
 		{
 			name: "namespace name does NOT contain suffix enforced by validating admission policy",
-			policy: withValidations([]admissionregistrationv1.Validation{
-				{
-					Expression: "object.metadata.name.endsWith(params.data.namespaceSuffix)",
-				},
-			}, withFailurePolicy(admissionregistrationv1.Fail, withParams(configParamKind(), withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", "validate-namespace-suffix-param"),
-			configMap: makeConfigParams("validate-namespace-suffix-param", map[string]string{
-				"namespaceSuffix": "k8s",
-			}),
 			namespace: &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
 					Name: "test-foo",
@@ -549,39 +587,9 @@ func Test_ValidateNamespace_WithConfigMapParams(t *testing.T) {
 			failureReason: metav1.StatusReasonInvalid,
 		},
 	}
-
 	for _, testcase := range testcases {
 		t.Run(testcase.name, func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
-			server, err := apiservertesting.StartTestServer(t, nil, []string{
-				"--enable-admission-plugins", "ValidatingAdmissionPolicy",
-			}, framework.SharedEtcd())
-			if err != nil {
-				t.Fatal(err)
-			}
-			defer server.TearDownFn()
-
-			config := server.ClientConfig
-
-			client, err := clientset.NewForConfig(config)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			if _, err := client.CoreV1().ConfigMaps("default").Create(context.TODO(), testcase.configMap, metav1.CreateOptions{}); err != nil {
-				t.Fatal(err)
-			}
-
-			policy := withWaitReadyConstraintAndExpression(testcase.policy)
-			if _, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}); err != nil {
-				t.Fatal(err)
-			}
-			if err := createAndWaitReady(t, client, testcase.policyBinding, nil); err != nil {
-				t.Fatal(err)
-			}
-
 			_, err = client.CoreV1().Namespaces().Create(context.TODO(), testcase.namespace, metav1.CreateOptions{})
-
 			checkExpectedError(t, err, testcase.err)
 			checkFailureReason(t, err, testcase.failureReason)
 		})
@@ -2131,16 +2139,31 @@ func Test_ValidatingAdmissionPolicy_ParamResourceDeletedThenRecreated(t *testing
 	}
 }
 
-// Test_CostLimitForValidation tests the cost limit set for a ValidatingAdmissionPolicy.
+// Test_CostLimitForValidation tests the cost limit set for a ValidatingAdmissionPolicy
+// with StrictCostEnforcementForVAP feature enabled.
 func Test_CostLimitForValidation(t *testing.T) {
+	resetPolicyRefreshInterval := generic.SetPolicyRefreshIntervalForTests(policyRefreshInterval)
+	defer resetPolicyRefreshInterval()
+	featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.StrictCostEnforcementForVAP, true)
+	server, err := apiservertesting.StartTestServer(t, nil, []string{
+		"--enable-admission-plugins", "ValidatingAdmissionPolicy",
+	}, framework.SharedEtcd())
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer server.TearDownFn()
+
+	config := server.ClientConfig
+	client, err := clientset.NewForConfig(config)
+	if err != nil {
+		t.Fatal(err)
+	}
+
 	testcases := []struct {
-		name                  string
-		policy                *admissionregistrationv1.ValidatingAdmissionPolicy
-		policyBinding         *admissionregistrationv1.ValidatingAdmissionPolicyBinding
-		namespace             *v1.Namespace
-		err                   string
-		failureReason         metav1.StatusReason
-		strictCostEnforcement bool
+		name          string
+		policy        *admissionregistrationv1.ValidatingAdmissionPolicy
+		err           string
+		failureReason metav1.StatusReason
 	}{
 		{
 			name: "With StrictCostEnforcementForVAP: Single expression exceeds per call cost limit for native library",
@@ -2149,15 +2172,8 @@ func Test_CostLimitForValidation(t *testing.T) {
 					Expression: "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(x, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(y, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z2, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z3, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z4, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z5, int('1'.find('[0-9]*')) < 100)))))))",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err:                   "operation cancelled: actual cost limit exceeded",
-			failureReason:         metav1.StatusReasonInvalid,
-			strictCostEnforcement: true,
+			err:           "operation cancelled: actual cost limit exceeded",
+			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
 			name: "With StrictCostEnforcementForVAP: Expression exceeds per call cost limit for extended library",
@@ -2167,15 +2183,8 @@ func Test_CostLimitForValidation(t *testing.T) {
 					Expression: "authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed() && authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed() && authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed()",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err:                   "operation cancelled: actual cost limit exceeded",
-			failureReason:         metav1.StatusReasonInvalid,
-			strictCostEnforcement: true,
+			err:           "operation cancelled: actual cost limit exceeded",
+			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
 			name: "With StrictCostEnforcementForVAP: Expression exceeds per call cost limit for extended library in variables",
@@ -2189,15 +2198,8 @@ func Test_CostLimitForValidation(t *testing.T) {
 					Expression: "variables.authzCheck",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err:                   "operation cancelled: actual cost limit exceeded",
-			failureReason:         metav1.StatusReasonInvalid,
-			strictCostEnforcement: true,
+			err:           "operation cancelled: actual cost limit exceeded",
+			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
 			name: "With StrictCostEnforcementForVAP: Expression exceeds per call cost limit for extended library in matchConditions",
@@ -2211,29 +2213,67 @@ func Test_CostLimitForValidation(t *testing.T) {
 					Expression: "true",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err:                   "operation cancelled: actual cost limit exceeded",
-			failureReason:         metav1.StatusReasonInvalid,
-			strictCostEnforcement: true,
+			err:           "operation cancelled: actual cost limit exceeded",
+			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
 			name:          "With StrictCostEnforcementForVAP: Expression exceeds per policy cost limit for extended library",
 			policy:        withValidations(generateValidationsWithAuthzCheck(29, "authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed()"), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
+			err:           "validation failed due to running out of cost budget, no further validation rules will be run",
+			failureReason: metav1.StatusReasonInvalid,
+		},
+	}
+	for _, testcase := range testcases {
+		t.Run(testcase.name, func(t *testing.T) {
+			policy := withWaitReadyConstraintAndExpression(testcase.policy)
+			if _, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}); err != nil {
+				t.Fatal(err)
+			}
+			policyBinding := makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", "")
+			if err := createAndWaitReady(t, client, policyBinding, nil); err != nil {
+				t.Fatal(err)
+			}
+
+			ns := &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
 					Name: "test-k8s",
 				},
-			},
-			err:                   "validation failed due to running out of cost budget, no further validation rules will be run",
-			failureReason:         metav1.StatusReasonInvalid,
-			strictCostEnforcement: true,
-		},
+			}
+			_, err = client.CoreV1().Namespaces().Create(context.TODO(), ns, metav1.CreateOptions{})
+			checkExpectedError(t, err, testcase.err)
+			checkFailureReason(t, err, testcase.failureReason)
+			if err := cleanupPolicy(t, client, policy, policyBinding); err != nil {
+				t.Fatalf("error while cleaning up policy and its bindings: %v", err)
+			}
+		})
+	}
+}
+
+// Test_CostLimitForValidationWithFeatureDisabled tests the cost limit set for a ValidatingAdmissionPolicy
+// with StrictCostEnforcementForVAP feature disabled.
+func Test_CostLimitForValidationWithFeatureDisabled(t *testing.T) {
+	resetPolicyRefreshInterval := generic.SetPolicyRefreshIntervalForTests(policyRefreshInterval)
+	defer resetPolicyRefreshInterval()
+	server, err := apiservertesting.StartTestServer(t, nil, []string{
+		"--enable-admission-plugins", "ValidatingAdmissionPolicy",
+	}, framework.SharedEtcd())
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer server.TearDownFn()
+
+	config := server.ClientConfig
+	client, err := clientset.NewForConfig(config)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	testcases := []struct {
+		name          string
+		policy        *admissionregistrationv1.ValidatingAdmissionPolicy
+		err           string
+		failureReason metav1.StatusReason
+	}{
 		{
 			name: "Without StrictCostEnforcementForVAP: Single expression exceeds per call cost limit for native library",
 			policy: withValidations([]admissionregistrationv1.Validation{
@@ -2241,15 +2281,8 @@ func Test_CostLimitForValidation(t *testing.T) {
 					Expression: "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(x, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(y, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z2, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z3, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z4, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].all(z5, int('1'.find('[0-9]*')) < 100)))))))",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			err:                   "operation cancelled: actual cost limit exceeded",
-			failureReason:         metav1.StatusReasonInvalid,
-			strictCostEnforcement: false,
+			err:           "operation cancelled: actual cost limit exceeded",
+			failureReason: metav1.StatusReasonInvalid,
 		},
 		{
 			name: "Without StrictCostEnforcementForVAP: Expression does not exceed per call cost limit for extended library",
@@ -2259,13 +2292,6 @@ func Test_CostLimitForValidation(t *testing.T) {
 					Expression: "authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed() && authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed() && authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed()",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			strictCostEnforcement: false,
 		},
 		{
 			name: "Without StrictCostEnforcementForVAP: Expression does not exceed per call cost limit for extended library in variables",
@@ -2279,56 +2305,36 @@ func Test_CostLimitForValidation(t *testing.T) {
 					Expression: "variables.authzCheck",
 				},
 			}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix"))))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			strictCostEnforcement: false,
 		},
 		{
-			name:          "Without StrictCostEnforcementForVAP: Expression does not exceed per policy cost limit for extended library",
-			policy:        withValidations(generateValidationsWithAuthzCheck(29, "authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed()"), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
-			policyBinding: makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", ""),
-			namespace: &v1.Namespace{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-k8s",
-				},
-			},
-			strictCostEnforcement: false,
+			name:   "Without StrictCostEnforcementForVAP: Expression does not exceed per policy cost limit for extended library",
+			policy: withValidations(generateValidationsWithAuthzCheck(29, "authorizer.group('apps').resource('deployments').subresource('status').namespace('test').name('backend').check('create').allowed()"), withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-namespace-suffix")))),
 		},
 	}
-	for _, testcase := range testcases {
+	for i, testcase := range testcases {
 		t.Run(testcase.name, func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.StrictCostEnforcementForVAP, testcase.strictCostEnforcement)
-
-			server, err := apiservertesting.StartTestServer(t, nil, []string{
-				"--enable-admission-plugins", "ValidatingAdmissionPolicy",
-			}, framework.SharedEtcd())
-			if err != nil {
-				t.Fatal(err)
-			}
-			defer server.TearDownFn()
-
-			config := server.ClientConfig
-
-			client, err := clientset.NewForConfig(config)
-			if err != nil {
-				t.Fatal(err)
-			}
 			policy := withWaitReadyConstraintAndExpression(testcase.policy)
 			if _, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}); err != nil {
 				t.Fatal(err)
 			}
-			if err := createAndWaitReady(t, client, testcase.policyBinding, nil); err != nil {
+			policyBinding := makeBinding("validate-namespace-suffix-binding", "validate-namespace-suffix", "")
+			if err := createAndWaitReady(t, client, policyBinding, nil); err != nil {
 				t.Fatal(err)
 			}
 
-			_, err = client.CoreV1().Namespaces().Create(context.TODO(), testcase.namespace, metav1.CreateOptions{})
+			nsName := fmt.Sprintf("test-%d-k8s", i)
+			ns := &v1.Namespace{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: nsName,
+				},
+			}
 
+			_, err = client.CoreV1().Namespaces().Create(context.TODO(), ns, metav1.CreateOptions{})
 			checkExpectedError(t, err, testcase.err)
 			checkFailureReason(t, err, testcase.failureReason)
+			if err := cleanupPolicy(t, client, policy, policyBinding); err != nil {
+				t.Fatalf("error while cleaning up policy and its bindings: %v", err)
+			}
 		})
 	}
 }
@@ -2344,33 +2350,65 @@ func generateValidationsWithAuthzCheck(num int, exp string) []admissionregistrat
 
 // TestCRDParams tests that a CustomResource can be used as a param resource for a ValidatingAdmissionPolicy.
 func TestCRDParams(t *testing.T) {
+	resetPolicyRefreshInterval := generic.SetPolicyRefreshIntervalForTests(policyRefreshInterval)
+	defer resetPolicyRefreshInterval()
+	featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
+	server, err := apiservertesting.StartTestServer(t, nil, []string{
+		"--enable-admission-plugins", "ValidatingAdmissionPolicy",
+	}, framework.SharedEtcd())
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer server.TearDownFn()
+
+	config := server.ClientConfig
+	client, err := clientset.NewForConfig(config)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	crd := versionedCustomResourceDefinition()
+	etcd.CreateTestCRDs(t, apiextensionsclientset.NewForConfigOrDie(server.ClientConfig), false, crd)
+	dynamicClient, err := dynamic.NewForConfig(config)
+	if err != nil {
+		t.Fatal(err)
+	}
+	gvr := schema.GroupVersionResource{
+		Group:    crd.Spec.Group,
+		Version:  crd.Spec.Versions[0].Name,
+		Resource: crd.Spec.Names.Plural,
+	}
+	crClient := dynamicClient.Resource(gvr)
+
+	resource := &unstructured.Unstructured{Object: map[string]interface{}{
+		"apiVersion": "awesome.bears.com/v1",
+		"kind":       "Panda",
+		"metadata": map[string]interface{}{
+			"name": "config-obj",
+		},
+		"spec": map[string]interface{}{
+			"nameCheck": "crd-test-k8s",
+		},
+	}}
+	_, err = crClient.Create(context.TODO(), resource, metav1.CreateOptions{})
+	if err != nil {
+		t.Fatalf("error creating %s: %s", gvr, err)
+	}
+
 	testcases := []struct {
 		name          string
-		resource      *unstructured.Unstructured
 		policy        *admissionregistrationv1.ValidatingAdmissionPolicy
-		policyBinding *admissionregistrationv1.ValidatingAdmissionPolicyBinding
 		namespace     *v1.Namespace
 		err           string
 		failureReason metav1.StatusReason
 	}{
 		{
 			name: "a rule that uses data from a CRD param resource does NOT pass",
-			resource: &unstructured.Unstructured{Object: map[string]interface{}{
-				"apiVersion": "awesome.bears.com/v1",
-				"kind":       "Panda",
-				"metadata": map[string]interface{}{
-					"name": "config-obj",
-				},
-				"spec": map[string]interface{}{
-					"nameCheck": "crd-test-k8s",
-				},
-			}},
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
 					Expression: "params.spec.nameCheck == object.metadata.name",
 				},
 			}, withNamespaceMatch(withParams(withCRDParamKind("Panda", "awesome.bears.com", "v1"), withFailurePolicy(admissionregistrationv1.Fail, makePolicy("test-policy"))))),
-			policyBinding: makeBinding("crd-policy-binding", "test-policy", "config-obj"),
 			namespace: &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
 					Name: "incorrect-name",
@@ -2381,22 +2419,11 @@ func TestCRDParams(t *testing.T) {
 		},
 		{
 			name: "a rule that uses data from a CRD param resource that does pass",
-			resource: &unstructured.Unstructured{Object: map[string]interface{}{
-				"apiVersion": "awesome.bears.com/v1",
-				"kind":       "Panda",
-				"metadata": map[string]interface{}{
-					"name": "config-obj",
-				},
-				"spec": map[string]interface{}{
-					"nameCheck": "crd-test-k8s",
-				},
-			}},
 			policy: withValidations([]admissionregistrationv1.Validation{
 				{
 					Expression: "params.spec.nameCheck == object.metadata.name",
 				},
 			}, withNamespaceMatch(withParams(withCRDParamKind("Panda", "awesome.bears.com", "v1"), withFailurePolicy(admissionregistrationv1.Fail, makePolicy("test-policy"))))),
-			policyBinding: makeBinding("crd-policy-binding", "test-policy", "config-obj"),
 			namespace: &v1.Namespace{
 				ObjectMeta: metav1.ObjectMeta{
 					Name: "crd-test-k8s",
@@ -2408,46 +2435,14 @@ func TestCRDParams(t *testing.T) {
 
 	for _, testcase := range testcases {
 		t.Run(testcase.name, func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
-			server, err := apiservertesting.StartTestServer(t, nil, []string{
-				"--enable-admission-plugins", "ValidatingAdmissionPolicy",
-			}, framework.SharedEtcd())
-			if err != nil {
-				t.Fatal(err)
-			}
-			defer server.TearDownFn()
-
-			config := server.ClientConfig
-
-			client, err := clientset.NewForConfig(config)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			crd := versionedCustomResourceDefinition()
-			etcd.CreateTestCRDs(t, apiextensionsclientset.NewForConfigOrDie(server.ClientConfig), false, crd)
-			dynamicClient, err := dynamic.NewForConfig(config)
-			if err != nil {
-				t.Fatal(err)
-			}
-			gvr := schema.GroupVersionResource{
-				Group:    crd.Spec.Group,
-				Version:  crd.Spec.Versions[0].Name,
-				Resource: crd.Spec.Names.Plural,
-			}
-			crClient := dynamicClient.Resource(gvr)
-			_, err = crClient.Create(context.TODO(), testcase.resource, metav1.CreateOptions{})
-			if err != nil {
-				t.Fatalf("error creating %s: %s", gvr, err)
-			}
-
 			policy := withWaitReadyConstraintAndExpression(testcase.policy)
 			if _, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}); err != nil {
 				t.Fatal(err)
 			}
 			// remove default namespace since the CRD is cluster-scoped
-			testcase.policyBinding.Spec.ParamRef.Namespace = ""
-			if err := createAndWaitReady(t, client, testcase.policyBinding, nil); err != nil {
+			policyBinding := makeBinding("crd-policy-binding", "test-policy", "config-obj")
+			policyBinding.Spec.ParamRef.Namespace = ""
+			if err := createAndWaitReady(t, client, policyBinding, nil); err != nil {
 				t.Fatal(err)
 			}
 
@@ -2455,6 +2450,10 @@ func TestCRDParams(t *testing.T) {
 
 			checkExpectedError(t, err, testcase.err)
 			checkFailureReason(t, err, testcase.failureReason)
+			if err := cleanupPolicy(t, client, policy, policyBinding); err != nil {
+				t.Fatalf("error while cleaning up policy and its bindings: %v", err)
+			}
+
 		})
 	}
 }
@@ -2554,6 +2553,21 @@ func TestBindingRemoval(t *testing.T) {
 // Test_ValidateSecondaryAuthorization tests a ValidatingAdmissionPolicy that performs secondary authorization checks
 // for both users and service accounts.
 func Test_ValidateSecondaryAuthorization(t *testing.T) {
+	resetPolicyRefreshInterval := generic.SetPolicyRefreshIntervalForTests(policyRefreshInterval)
+	defer resetPolicyRefreshInterval()
+	featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
+	server, err := apiservertesting.StartTestServer(t, nil, []string{
+		"--enable-admission-plugins", "ValidatingAdmissionPolicy",
+		"--authorization-mode=RBAC",
+		"--anonymous-auth",
+	}, framework.SharedEtcd())
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer server.TearDownFn()
+
+	// For test set up such as creating policies, bindings and RBAC rules.
+	adminClient := clientset.NewForConfigOrDie(server.ClientConfig)
 	testcases := []struct {
 		name             string
 		rbac             *rbacv1.PolicyRule
@@ -2606,7 +2620,7 @@ func Test_ValidateSecondaryAuthorization(t *testing.T) {
 		},
 	}
 
-	for _, testcase := range testcases {
+	for i, testcase := range testcases {
 		t.Run(testcase.name, func(t *testing.T) {
 			clients := map[string]func(t *testing.T, adminClient *clientset.Clientset, clientConfig *rest.Config, rules []rbacv1.PolicyRule) *clientset.Clientset{
 				"user":           secondaryAuthorizationUserClient,
@@ -2615,20 +2629,6 @@ func Test_ValidateSecondaryAuthorization(t *testing.T) {
 
 			for clientName, clientFn := range clients {
 				t.Run(clientName, func(t *testing.T) {
-					featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
-					server, err := apiservertesting.StartTestServer(t, nil, []string{
-						"--enable-admission-plugins", "ValidatingAdmissionPolicy",
-						"--authorization-mode=RBAC",
-						"--anonymous-auth",
-					}, framework.SharedEtcd())
-					if err != nil {
-						t.Fatal(err)
-					}
-					defer server.TearDownFn()
-
-					// For test set up such as creating policies, bindings and RBAC rules.
-					adminClient := clientset.NewForConfigOrDie(server.ClientConfig)
-
 					// Principal is always allowed to create and update namespaces so that the admission requests to test
 					// authorization expressions can be sent by the principal.
 					rules := []rbacv1.PolicyRule{{
@@ -2650,21 +2650,24 @@ func Test_ValidateSecondaryAuthorization(t *testing.T) {
 						testcase.extraAccountFn(t, adminClient, server.ClientConfig, extraRules)
 					}
 
+					policyName := fmt.Sprintf("%s-%s-%d", "validate-authz", clientName, i)
 					policy := withWaitReadyConstraintAndExpression(withValidations([]admissionregistrationv1.Validation{
 						{
 							Expression: testcase.expression,
 						},
-					}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy("validate-authz")))))
+					}, withFailurePolicy(admissionregistrationv1.Fail, withNamespaceMatch(makePolicy(policyName)))))
 					if _, err := adminClient.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}); err != nil {
 						t.Fatal(err)
 					}
-					if err := createAndWaitReady(t, adminClient, makeBinding("validate-authz-binding", "validate-authz", ""), nil); err != nil {
+					policyBindingName := fmt.Sprintf("%s-%s", policyName, "binding")
+					policyBinding := makeBinding(policyBindingName, policyName, "")
+					if err := createAndWaitReady(t, adminClient, policyBinding, nil); err != nil {
 						t.Fatal(err)
 					}
 
 					ns := &v1.Namespace{
 						ObjectMeta: metav1.ObjectMeta{
-							Name: "test-authz",
+							Name: fmt.Sprintf("%s-%s-%d", "test-authz", clientName, i),
 						},
 					}
 					_, err = client.CoreV1().Namespaces().Create(context.TODO(), ns, metav1.CreateOptions{})
@@ -2674,6 +2677,9 @@ func Test_ValidateSecondaryAuthorization(t *testing.T) {
 						expected = metav1.StatusReasonInvalid
 					}
 					checkFailureReason(t, err, expected)
+					if err := cleanupPolicy(t, adminClient, policy, policyBinding); err != nil {
+						t.Fatalf("error while cleaning up policy and its bindings: %v", err)
+					}
 				})
 			}
 		})
@@ -2765,65 +2771,240 @@ func TestCRDsOnStartup(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	// validate that namespaces starting with "test" are allowed
-	policyBinding := makeBinding("allowed-prefixes-binding", "allowed-prefixes", "test")
-	if err := createAndWaitReady(t, client, policyBinding, nil); err != nil {
+	// validate that namespaces starting with "test" are allowed
+	policyBinding := makeBinding("allowed-prefixes-binding", "allowed-prefixes", "test")
+	if err := createAndWaitReady(t, client, policyBinding, nil); err != nil {
+		t.Fatal(err)
+	}
+
+	doCheck := func(client clientset.Interface) {
+		if waitErr := wait.PollUntilContextTimeout(testContext, time.Millisecond*100, 3*time.Minute, true, func(ctx context.Context) (bool, error) {
+			disallowedNamespace := &v1.Namespace{
+				ObjectMeta: metav1.ObjectMeta{
+					GenerateName: "not-test-",
+				},
+			}
+
+			_, err = client.CoreV1().Namespaces().Create(testContext, disallowedNamespace, metav1.CreateOptions{})
+			if err == nil {
+				return false, nil
+			}
+
+			if strings.Contains(err.Error(), "not yet synced to use for admission") {
+				return false, nil
+			}
+
+			if strings.Contains(err.Error(), "failed to find resource referenced by paramKind") {
+				return false, nil
+			}
+
+			if !strings.Contains(err.Error(), "wrong prefix") {
+				return false, err
+			}
+
+			return true, nil
+		}); waitErr != nil {
+			t.Errorf("timed out waiting: %v", err)
+		}
+	}
+
+	// Show that the policy & binding are correctly working before restarting
+	// to use the paramKind and deliver an error
+	doCheck(client)
+	server.TearDownFn()
+
+	// Start the server.
+	server = apiservertesting.StartTestServerOrDie(
+		t,
+		&apiservertesting.TestServerInstanceOptions{},
+		[]string{
+			"--enable-admission-plugins", "ValidatingAdmissionPolicy",
+			"--authorization-mode=RBAC",
+			"--anonymous-auth",
+		},
+		etcdConfig)
+	defer server.TearDownFn()
+
+	// Now that the server is restarted, show again that the policy & binding are correctly working
+	client = clientset.NewForConfigOrDie(server.ClientConfig)
+
+	doCheck(client)
+
+}
+
+func TestAuthorizationDecisionCaching(t *testing.T) {
+	resetPolicyRefreshInterval := generic.SetPolicyRefreshIntervalForTests(policyRefreshInterval)
+	defer resetPolicyRefreshInterval()
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+	featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
+	var nChecks int
+	webhook := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		var review authorizationv1.SubjectAccessReview
+		if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+		}
+
+		review.Status.Allowed = true
+		if review.Spec.ResourceAttributes.Verb == "test" {
+			nChecks++
+			review.Status.Reason = fmt.Sprintf("%d", nChecks)
+		}
+
+		w.Header().Set("Content-Type", "application/json")
+		if err := json.NewEncoder(w).Encode(review); err != nil {
+			http.Error(w, err.Error(), http.StatusInternalServerError)
+		}
+	}))
+	defer webhook.Close()
+	kcfd, err := os.CreateTemp("", "kubeconfig-")
+	if err != nil {
+		t.Fatal(err)
+	}
+	func() {
+		defer func() {
+			if err := kcfd.Close(); err != nil {
+				t.Fatal(err)
+			}
+		}()
+		tmpl, err := template.New("kubeconfig").Parse(`
+apiVersion: v1
+kind: Config
+clusters:
+  - name: test-authz-service
+    cluster:
+      server: {{ .Server }}
+users:
+  - name: test-api-server
+current-context: webhook
+contexts:
+- context:
+    cluster: test-authz-service
+    user: test-api-server
+  name: webhook
+`)
+		if err != nil {
+			t.Fatal(err)
+		}
+		err = tmpl.Execute(kcfd, struct {
+			Server string
+		}{
+			Server: webhook.URL,
+		})
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+	_, config, teardown := framework.StartTestServer(ctx, t, framework.TestServerSetup{
+		ModifyServerRunOptions: func(options *options.ServerRunOptions) {
+			options.Admission.GenericAdmission.EnablePlugins = append(options.Admission.GenericAdmission.EnablePlugins, "ValidatingAdmissionPolicy")
+			if err = options.APIEnablement.RuntimeConfig.Set("api/all=true"); err != nil {
+				t.Fatal(err)
+			}
+
+			options.Authorization.Modes = []string{authzmodes.ModeWebhook}
+			options.Authorization.WebhookConfigFile = kcfd.Name()
+			options.Authorization.WebhookVersion = "v1"
+			// Bypass webhook cache to observe the policy plugin's cache behavior.
+			options.Authorization.WebhookCacheAuthorizedTTL = 0
+			options.Authorization.WebhookCacheUnauthorizedTTL = 0
+		},
+	})
+	defer teardown()
+
+	config = rest.CopyConfig(config)
+	config.Impersonate = rest.ImpersonationConfig{
+		UserName: "alice",
+		UID:      "1234",
+	}
+	client, err := clientset.NewForConfig(config)
+	if err != nil {
 		t.Fatal(err)
 	}
 
-	doCheck := func(client clientset.Interface) {
-		if waitErr := wait.PollUntilContextTimeout(testContext, time.Millisecond*100, 3*time.Minute, true, func(ctx context.Context) (bool, error) {
-			disallowedNamespace := &v1.Namespace{
+	for i, tc := range []struct {
+		name        string
+		validations []admissionregistrationv1.Validation
+	}{
+		{
+			name: "hit",
+			validations: []admissionregistrationv1.Validation{
+				{
+					Expression: "authorizer.requestResource.check('test').reason() == authorizer.requestResource.check('test').reason()",
+				},
+			},
+		},
+		{
+			name: "miss",
+			validations: []admissionregistrationv1.Validation{
+				{
+					Expression: "authorizer.requestResource.subresource('a').check('test').reason() == '1'",
+				},
+				{
+					Expression: "authorizer.requestResource.subresource('b').check('test').reason() == '2'",
+				},
+				{
+					Expression: "authorizer.requestResource.subresource('c').check('test').reason() == '3'",
+				},
+			},
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			policy := &admissionregistrationv1.ValidatingAdmissionPolicy{
 				ObjectMeta: metav1.ObjectMeta{
-					GenerateName: "not-test-",
+					Name: "test-authorization-decision-caching-policy",
+				},
+				Spec: admissionregistrationv1.ValidatingAdmissionPolicySpec{
+					MatchConstraints: &admissionregistrationv1.MatchResources{
+						ResourceRules: []admissionregistrationv1.NamedRuleWithOperations{
+							{
+								ResourceNames: []string{"test-authorization-decision-caching-namespace"},
+								RuleWithOperations: admissionregistrationv1.RuleWithOperations{
+									Operations: []admissionregistrationv1.OperationType{
+										admissionregistrationv1.Create,
+									},
+									Rule: admissionregistrationv1.Rule{
+										APIGroups:   []string{""},
+										APIVersions: []string{"v1"},
+										Resources:   []string{"namespaces"},
+									},
+								},
+							},
+						},
+					},
+					Validations: tc.validations,
 				},
 			}
 
-			_, err = client.CoreV1().Namespaces().Create(testContext, disallowedNamespace, metav1.CreateOptions{})
-			if err == nil {
-				return false, nil
+			policy, err = client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(ctx, withWaitReadyConstraintAndExpression(policy), metav1.CreateOptions{})
+			if err != nil {
+				t.Fatal(err)
 			}
 
-			if strings.Contains(err.Error(), "not yet synced to use for admission") {
-				return false, nil
+			policyBinding := makeBinding(policy.Name+"-binding", policy.Name, "")
+			if err := createAndWaitReady(t, client, policyBinding, nil); err != nil {
+				t.Fatal(err)
 			}
 
-			if strings.Contains(err.Error(), "failed to find resource referenced by paramKind") {
-				return false, nil
+			if _, err := client.CoreV1().Namespaces().Create(
+				ctx,
+				&v1.Namespace{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: fmt.Sprintf("test-authorization-decision-caching-namespace-%d", i),
+					},
+				},
+				metav1.CreateOptions{},
+			); err != nil {
+				if !apierrors.IsAlreadyExists(err) {
+					t.Fatal(err)
+				}
 			}
 
-			if !strings.Contains(err.Error(), "wrong prefix") {
-				return false, err
+			if err := cleanupPolicy(t, client, policy, policyBinding); err != nil {
+				t.Errorf("error while cleaning up policy and its bindings: %v", err)
 			}
-
-			return true, nil
-		}); waitErr != nil {
-			t.Errorf("timed out waiting: %v", err)
-		}
+		})
 	}
-
-	// Show that the policy & binding are correctly working before restarting
-	// to use the paramKind and deliver an error
-	doCheck(client)
-	server.TearDownFn()
-
-	// Start the server.
-	server = apiservertesting.StartTestServerOrDie(
-		t,
-		&apiservertesting.TestServerInstanceOptions{},
-		[]string{
-			"--enable-admission-plugins", "ValidatingAdmissionPolicy",
-			"--authorization-mode=RBAC",
-			"--anonymous-auth",
-		},
-		etcdConfig)
-	defer server.TearDownFn()
-
-	// Now that the server is restarted, show again that the policy & binding are correctly working
-	client = clientset.NewForConfigOrDie(server.ClientConfig)
-
-	doCheck(client)
-
 }
 
 type clientFn func(t *testing.T, adminClient *clientset.Clientset, clientConfig *rest.Config, rules []rbacv1.PolicyRule) *clientset.Clientset
@@ -2850,7 +3031,7 @@ func serviceAccountClient(namespace, name string) clientFn {
 	return func(t *testing.T, adminClient *clientset.Clientset, clientConfig *rest.Config, rules []rbacv1.PolicyRule) *clientset.Clientset {
 		clientConfig = rest.CopyConfig(clientConfig)
 		sa, err := adminClient.CoreV1().ServiceAccounts(namespace).Create(context.TODO(), &v1.ServiceAccount{ObjectMeta: metav1.ObjectMeta{Name: name}}, metav1.CreateOptions{})
-		if err != nil {
+		if err != nil && !apierrors.IsAlreadyExists(err) {
 			t.Fatal(err)
 		}
 		uid := sa.UID
@@ -2951,6 +3132,39 @@ func createAndWaitReadyNamespacedWithWarnHandler(t *testing.T, client clientset.
 	return nil
 }
 
+func cleanupPolicy(t *testing.T, client clientset.Interface, policy *admissionregistrationv1.ValidatingAdmissionPolicy, binding *admissionregistrationv1.ValidatingAdmissionPolicyBinding) error {
+	err := client.AdmissionregistrationV1().ValidatingAdmissionPolicyBindings().Delete(context.TODO(), binding.Name, metav1.DeleteOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if waitErr := wait.PollUntilContextTimeout(context.TODO(), time.Second, time.Minute, true, func(ctx context.Context) (bool, error) {
+		_, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicyBindings().Get(context.TODO(), binding.Name, metav1.GetOptions{})
+		if apierrors.IsNotFound(err) {
+			return true, nil
+		}
+		return false, err
+	}); waitErr != nil {
+		t.Fatalf("timed out waiting for policy binding to be cleaned up: %v", err)
+	}
+
+	if err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Delete(context.TODO(), policy.Name, metav1.DeleteOptions{}); err != nil {
+		t.Fatal(err)
+	}
+
+	if waitErr := wait.PollUntilContextTimeout(context.TODO(), time.Second, time.Minute, true, func(ctx context.Context) (bool, error) {
+		_, err := client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Get(context.TODO(), policy.Name, metav1.GetOptions{})
+		if apierrors.IsNotFound(err) {
+			return true, nil
+		}
+		return false, nil
+	}); waitErr != nil {
+		t.Fatalf("timed out waiting for policy to be cleaned up: %v", err)
+	}
+
+	return nil
+}
+
 func withMatchNamespace(binding *admissionregistrationv1.ValidatingAdmissionPolicyBinding, ns string) *admissionregistrationv1.ValidatingAdmissionPolicyBinding {
 	binding.Spec.MatchResources = &admissionregistrationv1.MatchResources{
 		NamespaceSelector: &metav1.LabelSelector{
@@ -3354,167 +3568,3 @@ rules:
         resources: ["configmaps"]
 `
 )
-
-func TestAuthorizationDecisionCaching(t *testing.T) {
-	for _, tc := range []struct {
-		name        string
-		validations []admissionregistrationv1.Validation
-	}{
-		{
-			name: "hit",
-			validations: []admissionregistrationv1.Validation{
-				{
-					Expression: "authorizer.requestResource.check('test').reason() == authorizer.requestResource.check('test').reason()",
-				},
-			},
-		},
-		{
-			name: "miss",
-			validations: []admissionregistrationv1.Validation{
-				{
-					Expression: "authorizer.requestResource.subresource('a').check('test').reason() == '1'",
-				},
-				{
-					Expression: "authorizer.requestResource.subresource('b').check('test').reason() == '2'",
-				},
-				{
-					Expression: "authorizer.requestResource.subresource('c').check('test').reason() == '3'",
-				},
-			},
-		},
-	} {
-		t.Run(tc.name, func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)
-
-			ctx, cancel := context.WithCancel(context.TODO())
-			defer cancel()
-
-			var nChecks int
-			webhook := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-				var review authorizationv1.SubjectAccessReview
-				if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
-					http.Error(w, err.Error(), http.StatusBadRequest)
-				}
-
-				review.Status.Allowed = true
-				if review.Spec.ResourceAttributes.Verb == "test" {
-					nChecks++
-					review.Status.Reason = fmt.Sprintf("%d", nChecks)
-				}
-
-				w.Header().Set("Content-Type", "application/json")
-				if err := json.NewEncoder(w).Encode(review); err != nil {
-					http.Error(w, err.Error(), http.StatusInternalServerError)
-				}
-			}))
-			defer webhook.Close()
-
-			kcfd, err := os.CreateTemp("", "kubeconfig-")
-			if err != nil {
-				t.Fatal(err)
-			}
-			func() {
-				defer kcfd.Close()
-				tmpl, err := template.New("kubeconfig").Parse(`
-apiVersion: v1
-kind: Config
-clusters:
-  - name: test-authz-service
-    cluster:
-      server: {{ .Server }}
-users:
-  - name: test-api-server
-current-context: webhook
-contexts:
-- context:
-    cluster: test-authz-service
-    user: test-api-server
-  name: webhook
-`)
-				if err != nil {
-					t.Fatal(err)
-				}
-				err = tmpl.Execute(kcfd, struct {
-					Server string
-				}{
-					Server: webhook.URL,
-				})
-				if err != nil {
-					t.Fatal(err)
-				}
-			}()
-
-			client, config, teardown := framework.StartTestServer(ctx, t, framework.TestServerSetup{
-				ModifyServerRunOptions: func(options *options.ServerRunOptions) {
-					options.Admission.GenericAdmission.EnablePlugins = append(options.Admission.GenericAdmission.EnablePlugins, "ValidatingAdmissionPolicy")
-					options.APIEnablement.RuntimeConfig.Set("api/all=true")
-
-					options.Authorization.Modes = []string{authzmodes.ModeWebhook}
-					options.Authorization.WebhookConfigFile = kcfd.Name()
-					options.Authorization.WebhookVersion = "v1"
-					// Bypass webhook cache to observe the policy plugin's cache behavior.
-					options.Authorization.WebhookCacheAuthorizedTTL = 0
-					options.Authorization.WebhookCacheUnauthorizedTTL = 0
-				},
-			})
-			defer teardown()
-
-			policy := &admissionregistrationv1.ValidatingAdmissionPolicy{
-				ObjectMeta: metav1.ObjectMeta{
-					Name: "test-authorization-decision-caching-policy",
-				},
-				Spec: admissionregistrationv1.ValidatingAdmissionPolicySpec{
-					MatchConstraints: &admissionregistrationv1.MatchResources{
-						ResourceRules: []admissionregistrationv1.NamedRuleWithOperations{
-							{
-								ResourceNames: []string{"test-authorization-decision-caching-namespace"},
-								RuleWithOperations: admissionregistrationv1.RuleWithOperations{
-									Operations: []admissionregistrationv1.OperationType{
-										admissionregistrationv1.Create,
-									},
-									Rule: admissionregistrationv1.Rule{
-										APIGroups:   []string{""},
-										APIVersions: []string{"v1"},
-										Resources:   []string{"namespaces"},
-									},
-								},
-							},
-						},
-					},
-					Validations: tc.validations,
-				},
-			}
-
-			policy, err = client.AdmissionregistrationV1().ValidatingAdmissionPolicies().Create(ctx, withWaitReadyConstraintAndExpression(policy), metav1.CreateOptions{})
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			if err := createAndWaitReady(t, client, makeBinding(policy.Name+"-binding", policy.Name, ""), nil); err != nil {
-				t.Fatal(err)
-			}
-
-			config = rest.CopyConfig(config)
-			config.Impersonate = rest.ImpersonationConfig{
-				UserName: "alice",
-				UID:      "1234",
-			}
-			client, err = clientset.NewForConfig(config)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			if _, err := client.CoreV1().Namespaces().Create(
-				ctx,
-				&v1.Namespace{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: "test-authorization-decision-caching-namespace",
-					},
-				},
-				metav1.CreateOptions{},
-			); err != nil {
-				t.Fatal(err)
-			}
-		})
-	}
-}