Change:
Per the comment in this PR conformance tests will regress because of a newly Runtime Class GA v1 API is not covered with tests. Full comment: Full comment: /assign More links:
/sig node /triage accepted
KEP:
New to conformance tests. So the idea is to mark tests (or create new ones) that would call each of these methods at lease once? So if this test will be marked as conformance, it will cover and . Is there any additional requirements for the conformance tests? Maybe with the number of side effects of that test or a special execution environment?
Yes. You can make a single test that runs objects through CRUD operations. For the deletecollection one, I'd suggest labeling the objects under test and scoping the deletecollection to your test label to avoid breaking the cluster you're running on. See as an example
Great, will be working on it! Also just chatted with about it.
Just to get clarity on this - is this considered a blocker for Confirming bc grad criteria for it says "No release blocking feedback for API and functionality" should these tests be required to consider this stable? As a note: Code Freeze means that required tests must also be finished:
Here's possibly an easier link to track: !
This seems like something that should be tested generically, or at the very least have a table-driven test that covers all (or most) GA APIs.
! Just a quick note to show that currently the new tests are hitting all endpoints. I'd go ahead and create the Promotion PR in order to get any necessary feedback from #sig-arch
/reopen re: I disagree with the idea that a single test covering CRUD operations means RuntimeClass is now conformance tested. Generic CRUD on a resource's API is not sufficient to describe the intended behavior of the resource. That kind of test is at best verifying that api machinery is working. A vendor would able to pass this test, without implementing any of the behaviors runtimeclass enables. What does do? What kinds of behaviors are visible to an end-user? Are any of those tests in Conformance?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm not familiar enough with runtimeclass to know whether these would be a good start:
Thank you for reopening. I agree with this in principle. But I really would appreciate some guidance here. Scheduling tests that takes runtime handler into account are great. However they don't hit any endpoints. I asked and I'm not clear whether those needs to be covered as they are not touching an API. As for actual scheduling, one limitation is posted . Which makes some other tests impossible to include to conformance. I next week, but there may not be all the right people there as it's a Thanksgiving short week. Do you have a suggestion?
I agree that exercising specific CRI implementations is not the place of Conformance testing. Conformance testing is intended to exercise end-user visible behaviors of a Kubernetes cluster, which basically means whatever you can see and do at the Control Plane API level (ref: ) The challenge we face with Conformance testing is that we don't have an easy way to enumerate and measure coverage of behaviors. We experimented with listing behaviors in .yaml so that humans and machines could populate and approve them independent of test implementation, but ultimately abandoned that effort due to its complexity (ref: ) So we use API coverage as an easily machine-measurable/enforceable proxy. Noticing the coverage was at 0% should have been a sign that we should look at the existing test coverage of RuntimeClass and decide which tests were eligible for promotion to Conformance, and whether any new tests needed to be written. In theory a solid test plan for a KEP would include a list of behaviors that need to be tested. That's not immediately clear to me from the KEP for RuntimeClass (ref: ) Based on my read of the KEP, some valid tests might be: tests that exercise the runtimeclass admission controller creating pods with a non-existent runtimeclass creating pods with an existing runtimeclass creating pods with scheduling rules that would clash with runtimeclass scheduling rules creating pods with scheduling rules that would merge with runtimeclass scheduling rules Tests that would be disallowed by conformance requirements because they would require pre-installation/configuration of the cluster under test: running pods with a runtimeclass (unless there's some way to create a functional "dummy" runtimeclass without touching kubelets) running pods with a runtimeclass with a specific handler such as containerd, crio, etc. and expecting them to run
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
sorry for the delay /remove-lifecycle stale
/area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
patch, delete, and deletecollection.' release: v1.20 file: test/e2e/common/node/runtimeclass.go <ins> - testname: Pod with the deleted RuntimeClass is rejected. codename: '[sig-node] RuntimeClass should reject a Pod requesting a deleted RuntimeClass [NodeConformance] [Conformance]' description: Pod requesting the deleted RuntimeClass must be rejected. release: v1.20 file: test/e2e/common/node/runtimeclass.go - testname: Pod with the non-existing RuntimeClass is rejected. codename: '[sig-node] RuntimeClass should reject a Pod requesting a non-existent RuntimeClass [NodeConformance] [Conformance]' description: The Pod requesting the non-existing RuntimeClass must be rejected. release: v1.20 file: test/e2e/common/node/runtimeclass.go - testname: RuntimeClass Overhead field must be respected. codename: '[sig-node] RuntimeClass should schedule a Pod requesting a RuntimeClass and initialize its Overhead [NodeConformance] [Conformance]' description: The Pod requesting the existing RuntimeClass must be scheduled. This test doesn't validate that the Pod will actually start because this functionality depends on container runtime and preconfigured handler. Runtime-specific functionality is not being tested here. release: v1.24 file: test/e2e/common/node/runtimeclass.go - testname: Can schedule a pod requesting existing RuntimeClass. codename: '[sig-node] RuntimeClass should schedule a Pod requesting a RuntimeClass without PodOverhead [NodeConformance] [Conformance]' description: The Pod requesting the existing RuntimeClass must be scheduled. This test doesn't validate that the Pod will actually start because this functionality depends on container runtime and preconfigured handler. Runtime-specific functionality is not being tested here. release: v1.20 file: test/e2e/common/node/runtimeclass.go </ins> - testname: Secrets, pod environment field codename: '[sig-node] Secrets should be consumable from pods in env vars [NodeConformance] [Conformance]'

Testname: Pod with the non-existing RuntimeClass is rejected. Description: The Pod requesting the non-existing RuntimeClass must be rejected. */ <del> ginkgo.It("should reject a Pod requesting a non-existent RuntimeClass [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should reject a Pod requesting a non-existent RuntimeClass [NodeConformance]", func() { </ins> rcName := f.Namespace.Name + "-nonexistent" expectPodRejection(f, e2enode.NewRuntimeClassPod(rcName)) })

depends on container runtime and preconfigured handler. Runtime-specific functionality is not being tested here. */ <del> ginkgo.It("should schedule a Pod requesting a RuntimeClass without PodOverhead [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should schedule a Pod requesting a RuntimeClass without PodOverhead [NodeConformance]", func() { </ins> rcName := createRuntimeClass(f, "preconfigured-handler", e2enode.PreconfiguredRuntimeClassHandler, nil) defer deleteRuntimeClass(f, rcName) pod := f.PodClient().Create(e2enode.NewRuntimeClassPod(rcName))

depends on container runtime and preconfigured handler. Runtime-specific functionality is not being tested here. */ <del> ginkgo.It("should schedule a Pod requesting a RuntimeClass and initialize its Overhead [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should schedule a Pod requesting a RuntimeClass and initialize its Overhead [NodeConformance]", func() { </ins> rcName := createRuntimeClass(f, "preconfigured-handler", e2enode.PreconfiguredRuntimeClassHandler, &nodev1.Overhead{ PodFixed: v1.ResourceList{ v1.ResourceName(v1.ResourceCPU):    resource.MustParse("10m"),

Testname: Pod with the deleted RuntimeClass is rejected. Description: Pod requesting the deleted RuntimeClass must be rejected. */ <del> ginkgo.It("should reject a Pod requesting a deleted RuntimeClass [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should reject a Pod requesting a deleted RuntimeClass [NodeConformance]", func() { </ins> rcName := createRuntimeClass(f, "delete-me", "runc", nil) rcClient := f.ClientSet.NodeV1().RuntimeClasses()