Change:
Description: The ValidatingAdmissionPolicy should type check a CRD. */ <del> framework.ConformanceIt("should type check a CRD", func(ctx context.Context) { </del> <ins> framework.It("should type check a CRD", func(ctx context.Context) { </ins> crd := crontabExampleCRD() crd.Spec.Group = "stable." + f.UniqueName crd.Name = crd.Spec.Names.Plural + "." + crd.Spec.Group

Discussion:
<img width="1015" alt="image" src="" /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig api-machinery
/priority critical-urgent Looking at the last green and first red shows the following: xref : /assign
hrmm, looks super solid in
yea something screwy going on with the kubeadm jobs :(
my main concern what that this was a test ..
kinder or kind jobs have been mostly solid across the board for a few years. if something is failing in kind / kinder then likely it would also fail in resource constraint scenarios. sorry but my 2c are that this is worth investigating by owning sigs
i don't know what else could be the cause here. kubeadm is crd agnostic.
completely agree this needs resolution, I'm just confused since is a really similar setup and looks completely solid The test waits until the status is set, so even a resource-constrained env should just wait a little longer ... the failures I see are bailing out of that loop after only one wait (105ms), so something is turning that field non-nil without populating the expected results I'm also pretty confused
the diff between the two jobs (kinder/kind) reveals that the kind job has much higher resource limits/requests (for CPU) full diff: <details</details
i'm not familiar with this area of the code. it does appear to be waiting for the TypeChecking struct to be populated, but not for the Warnings slice to be populated. logically (without trying to find the code): if TypeChecking is expected to be populated with Warnings during a Get call after the Create call, we are catching a flake bug. if the Warnings in the TypeChecking are populated async of the TypeChecking then we are catching a test flake and the test must be updated to wait for the Warnings too.
well, the code is here: it appears that the only way for this to happen is if the Check() call returns nil warnings under "some conditions". the problem is not reproducible in the unit test here if the interval is set to a lower value <= 100ms and tested with the tool. but that's with a fake client.

this extra err check seems redundant: it might be worth temporary adding a print of the crd and policy before this line: then allow this test to flake again and inspect the results.
/milestone v1.30
Thanks for raising the issue and thanks for the investigation! I don't know exactly why the solid test in kind would be flaky in kubeadm by a quick scan but agree it needs attention.(The tests look less flaky though..) /assign the type checking code owner. Would you mind adding a printout as suggested above and see if we could get more info? Thanks!
note, both kind and kinder (from the compared jobs) use kubeadm.
and i wonder if this is an informer cache issue.
Same guess as failed for missing event for service patch since 03/06(similar time). [Not sure; just guess]
shows that the test failed for a different reason. failure in ci-kubernetes-e2e-ec2-alpha-enabled-default: from 03/04 to 03/06.
/remove-kind failing-test /kind flake
the API field doc indicates a non-nil value means type checking is complete: It doesn't look like the type checker has any way to signal that it was lacking some information needed to complete type checking, and will record an empty set of warnings if it is: are there any scenarios where a laggy REST mapper or informer can make the controller think it completed type checking but be missing some info that makes it produce an empty list of warnings? I think our options are: the test to wait for non-empty warnings in the status (not just non-nil typeChecking) the controller to not write an empty typeChecking stanza if it fails to type check in some way option 2 seems better for observers, but I don't know how invasive that is
option 2 feels like we are creating a hole? (folks will see warnings sometimes and sometimes they don't?)
Option 2 might cause confusion especially when deleting existing warnings from typechecking field.

(We will remove the typechecking warning if observed the fix in schema)
I'm not suggesting deleting existing warnings, but it appears we are currently reporting an empty typeChecking stanza when we have not actually finished type checking... that doesn't match the API docs and is likely confusing to API clients (it's confusing our own e2e test we wrote based on the documented behavior)
Ignoring errors while finishing the type checking is intended behavior defined by the original KEP. Can we go back to change the tests to do a retry loop until the CRD appears in the RESTMapper? The root problem of laggy RESTMapper seems out of the scope of this test case.
if is fixing the test then it's masking a state not guaranteed by the field comment: i guess already explained that above.
The type checking is completed with this field set to non-nil. The problem is that the type checking finishes with error, and it cause some of the referred types to be ignored.
should this error be surfaced to the user and does that need an e2e test?
No. The KEP defines that errors will be ignored. However, there is a next step to make the type checker reacts on CRD changes, which handles the case when the CRD schema is late for the type checking.
i think users might be confused by this behavior (the same way people on this thread are) even if it's documented in a KEP, so a field comment update feels worth it, at minimum.
The current type checking is informational and only works on ValidatingAdmissionPolicy. If the CRD is created after type checking or CRD created but not showing in schema watcher yet, the error would be ignored. And it will not retry as long as ValidatingAdmissionPolicy remains unchanged. There is a PR for adding retry with schema resolver raised in which will solve the issue entirely. In current case, the guess would be schema watcher is slow so it didn't have crd info yet when performing typechecking. The temporary fix by in will resolve the test flaky by adding manually retry. The field comment of makes sense to me since it indicates the completion of the type checking but not handling the case where the CRD has not been created or created but not showing in schema watcher yet. FYI The says I agree it might not fit in conformance test. Not sure about the conformance test criteria.
cc
/triage accepted