Change:
I'm using ValidatingAdmissionPolicy(VAP) with setting my own CRD to paramKind in my on-premise cluster. It works fine, but once I restarted one of the kube-apiserver instances it changed to reject both valid and invalid requests with the error message like: VAP using CRDs as paramKind should continue to work fine even when one of kube-apiserver instances is restarted. You can reproduce it locally using kind cluster by following procedure. <details</detailsI recovered this failure situation by mutating field of all existing VAPs and running reconciliation of this field explicitly. I've also tried to mutate , but it is not enough. Without the above operation, all requests are rejected. I might be able to avoid this error by setting to Allow, but I don't want to do that to enforce this validation check. v1.28.3, v1.29.0 <details</detailsN/A N/A N/A N/A N/A
/sig api-machinery
/triage accepted
We are experiencing the same issue with Kubernetes v1.27.9, but it is very flaky and does not occur all the time. Edit 2: it's totally flaky and cannot be mitigated with the feature gate . Sometimes a single restart of the kube-apiserver fixes this bug again, sometimes it needs multiple restarts.
cc I don't think we should go to GA without a fix for this.
How long does this condition last for you? The param CRD is checked when the policy is resynced as you said. The controller is written so that if an error is returned from looking up the param resource it retries again later, so the manual resync shouldn't be necessary. But it does use exponential backoff: I'd be curious if you have more logs, specifically around lines from the controller automatic resync that include "error syncing" Additionally, the RESTMapper used to perform this check is refreshed every 30 seconds, so there could be up to a 30s delay at least after a CRD is that it becomes visible (assuming no rate limit):
/assign Thanks for taking this! Thanks for raising the issue. Would you mind sharing more info on this one especially how long/often this happens, any log to share, is the issue still there after waiting for longer, etc after Alex shared more details above? Thank you!
Thank you for your comments! I have reproduced this problem again with the procedure as I described. As you know, this problem is flaky and it recurred 3 times in 5 trials. When it recurs it continues for over an hour. I’m not sure how long it will continue, I'm currently running the test for a long time to investigate it. I will let you know when I get the results. (Probably, it will continue until next resync will be performed.) Here is the log around the when the problem is recurring. If needed, I will get more detailed log and share it with you.
I kept running the test for over a day, and it continued to happen throughout the test. I tried to recover it by patching the 'spec.paramKind' field like following, but it didn’t work in the 1.29 kind environment. (This worked in our 1.28.3 on-premise cluster, so there might be some difference between these versions.). After all, I deleted and recreated the validatingAdmissionPolicy then it recovered to execute validation correctly.
Thanks for your detailed report! From this I think I may have found the source of the issue. It was curious to me that your logs only contained a single occurrance of: This indicates to me that it was not retrying correctly. Snooping around the code I see this line to the top of : This short circuits the resync since the spec did not change. I was able to reproduce your bug report in an integration test, and with a small code change I now see the policy syncing retried multiple times (with exponential backoff) until success after restarting the apiserver: code change and test here:
thanks a lot for the fix. Unfortunately, it will take a lot of time until this arrives to the distro that we are using. As we already rely on VAP for some policy enforcement, we would like to understand if there would be any simple workaround that can help when this issue happens (apart from restart the API several times when it gets fixed...).
backport PR is already created, see
thanks. Yes, just realized it after I wrote my comment. Until that arrives to the distros, what would be the simplet/best way to "trigger" a sync manually to "get rid of the issue" when it happens? Seems deleting the VAP, VAPB and CRD and re-apply them removes the issue... Is there any better way?

Discussion:
celmetrics.Metrics.ObserveDefinition(context.TODO(), "active", "deny") } <del> // Skip reconcile if the spec of the definition is unchanged if info.lastReconciledValue != nil && definition != nil && </del> <ins> // Skip reconcile if the spec of the definition is unchanged and had a // successful previous sync if info.configurationError == nil && info.lastReconciledValue != nil && definition != nil && </ins> apiequality.Semantic.DeepEqual(info.lastReconciledValue.Spec, definition.Spec) { return nil }

} } <ins> func TestCRDsOnStartup(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)() testContext, testCancel := context.WithCancel(context.Background()) defer testCancel() // Start server and create CRD, and validatingadmission policy and binding etcdConfig := framework.SharedEtcd() server := apiservertesting.StartTestServerOrDie(t, nil, []string{ "--enable-admission-plugins", "ValidatingAdmissionPolicy", "--authorization-mode=RBAC", "--anonymous-auth", }, etcdConfig) client := clientset.NewForConfigOrDie(server.ClientConfig) dynamicClient := dynamic.NewForConfigOrDie(server.ClientConfig) apiextclient := apiextensionsclientset.NewForConfigOrDie(server.ClientConfig) myCRD := &apiextensionsv1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{ Name: "foos.cr.bar.com", }, Spec: apiextensionsv1.CustomResourceDefinitionSpec{ Group: "cr.bar.com", Scope: apiextensionsv1.NamespaceScoped, Names: apiextensionsv1.CustomResourceDefinitionNames{ Plural: "foos", Kind:   "Foo", }, Versions: []apiextensionsv1.CustomResourceDefinitionVersion{ { Name:    "v1", Served:  true, Storage: true, Schema:  fixtures.AllowAllSchema(), }, }, }, } // Create a bunch of fake CRDs to make the initial startup sync take a long time for i := 0; i < 100; i++ { crd := myCRD.DeepCopy() crd.Name = fmt.Sprintf("foos%d.cr.bar.com", i) crd.Spec.Names.Plural = fmt.Sprintf("foos%d", i) crd.Spec.Names.Kind = fmt.Sprintf("Foo%d", i) if _, err := apiextclient.ApiextensionsV1().CustomResourceDefinitions().Create(context.Background(), crd, metav1.CreateOptions{}); err != nil { t.Fatal(err) } } etcd.CreateTestCRDs(t, apiextclient, false, myCRD) crdGVK := schema.GroupVersionKind{ Group:   "cr.bar.com", Version: "v1", Kind:    "Foo", } crdGVR := crdGVK.GroupVersion().WithResource("foos") param := &unstructured.Unstructured{ Object: map[string]interface{}{ "metadata": map[string]interface{}{ "name":      "test", "namespace": "default", }, "foo": "bar", }, } param.GetObjectKind().SetGroupVersionKind(crdGVK) if _, err := dynamicClient.Resource(crdGVR).Namespace("default").Create(context.TODO(), param, metav1.CreateOptions{}); err != nil { t.Fatal(err) } policy := withValidations([]admissionregistrationv1beta1.Validation{ { Expression: "object.metadata.name.startsWith(params.metadata.name)", Message:    "wrong prefix", }, }, withParams(withCRDParamKind(crdGVK.Kind, crdGVK.Group, crdGVK.Version), withNamespaceMatch(withFailurePolicy(admissionregistrationv1beta1.Fail, makePolicy("allowed-prefixes"))))) policy = withWaitReadyConstraintAndExpression(policy) _, err := client.AdmissionregistrationV1beta1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}) if err != nil { t.Fatal(err) } // validate that namespaces starting with "test" are allowed policyBinding := makeBinding("allowed-prefixes-binding", "allowed-prefixes", "test") if err := createAndWaitReady(t, client, policyBinding, nil); err != nil { t.Fatal(err) } doCheck := func(client clientset.Interface) { if waitErr := wait.PollUntilContextTimeout(testContext, time.Millisecond*100, 3*time.Minute, true, func(ctx context.Context) (bool, error) { disallowedNamespace := &v1.Namespace{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "not-test-", }, } _, err = client.CoreV1().Namespaces().Create(testContext, disallowedNamespace, metav1.CreateOptions{}) if err == nil { return false, nil } if strings.Contains(err.Error(), "not yet synced to use for admission") { return false, nil } if strings.Contains(err.Error(), "failed to find resource referenced by paramKind") { return false, nil } if !strings.Contains(err.Error(), "wrong prefix") { return false, err } return true, nil }); waitErr != nil { t.Errorf("timed out waiting: %v", err) } } // Show that the policy & binding are correctly working before restarting // to use the paramKind and deliver an error doCheck(client) server.TearDownFn() // Start the server. server = apiservertesting.StartTestServerOrDie( t, &apiservertesting.TestServerInstanceOptions{ SkipHealthzCheck: true, }, []string{ "--enable-admission-plugins", "ValidatingAdmissionPolicy", "--authorization-mode=RBAC", "--anonymous-auth", }, etcdConfig) defer server.TearDownFn() // Now that the server is restarted, show again that the policy & binding are correctly working client = clientset.NewForConfigOrDie(server.ClientConfig) doCheck(client) } </ins> type clientFn func(t *testing.T, adminClient *clientset.Clientset, clientConfig *rest.Config, rules []rbacv1.PolicyRule) *clientset.Clientset func secondaryAuthorizationUserClient(t *testing.T, adminClient *clientset.Clientset, clientConfig *rest.Config, rules []rbacv1.PolicyRule) *clientset.Clientset {