Change:
What happened: Ran a conformance test run (using Sonobuoy) on One of the taint based tests a NoExecute taint to a node which evicted all workloads including the Sonobuoy workload itself. If this had been a prod workload, numerous prod workloads could have been impacted. What you expected to happen: Test run as normal with reasonable results. Able to run conformance tests with minimal impact on a running cluster. How to reproduce it (as minimally and precisely as possible): on a v1.16.0 cluster. Anything else we need to know?: Environment: Kubernetes version (use ): /area conformance
/sig testing /sig architecture /sig scheduling
Summarizing the many conversations on the PR and in Slack regarding this: We wish we had identified the lack of a tag sooner and could have taken better action. Since the release is 2 days away and our own tooling currently prohibits us from marking tests as both and , we've decided to let the release proceed with this "bug". Risk: Someone running the conformance tests may unexpectedly have their workloads evicted Mitigation: Sonobuoy v0.16.0 will provide the capabilities to the user to run a "conformance-lite" (name TBD) run if they'd like to run the conformance tests against a cluster with prod workloads. This way end-users can still be reasonably sure of their cluster's continued well-being. This mode will manually skip the tests in question until they can be more appropriately marked as Sonobuoy will also provide, consistent with past behavior, the means to run the full conformance suite including these disruptive tests. Eventual solution: Sonobuoy will, most likely, need to continue offering a "non-disruptive-conformance" mode to aid in end user testing but the regexp will change and should be able to just skip tests K8s tooling needs to be tweaked to allow for tests to be marked as The tag should be to these tests.

Discussion:
totalConfTests, totalLegacyTests, missingComments int // If a test name contains any of these tags, it is ineligble for promotion to conformance <del> regexIneligibleTags = regexp.MustCompile(`[(Alpha|Disruptive|Feature:[^]]+|Flaky)]`) </del> <ins> regexIneligibleTags = regexp.MustCompile(`[(Alpha|Feature:[^]]+|Flaky)]`) </ins> ) const regexDescribe = "Describe|KubeDescribe|SIGDescribe"

"", }, { <del> "a test case with valid tags [LinuxOnly] [NodeConformance] [Serial]", </del> <ins> "a test case with valid tags [LinuxOnly] [NodeConformance] [Serial] [Disruptive]", </ins> "", }, {

"[Flaky]", }, { <del> "a disruptive test case that is invalid [Disruptive]", "[Disruptive]", }, { </del> "a feature test case that is invalid [Feature:Awesome]", "[Feature:Awesome]", },

"[Alpha]", }, { <del> "a test case with multiple invalid tags [Flaky][Disruptive] [Feature:Awesome] [Alpha]", "[Flaky],[Disruptive],[Feature:Awesome],[Alpha]", </del> <ins> "a test case with multiple invalid tags [Flaky] [Feature:Awesome] [Alpha]", "[Flaky],[Feature:Awesome],[Alpha]", </ins> }, { <del> "[sig-awesome] [Disruptive] a test case with valid and invalid tags [Alpha] [Serial] [Flaky]", "[Disruptive],[Alpha],[Flaky]", </del> <ins> "[sig-awesome] [Alpha] [Disruptive] a test case with valid and invalid tags [Serial] [Flaky]", "[Alpha],[Flaky]", </ins> }, } for i, tc := range testCases {

Description: The Pod with toleration timeout scheduled on a tainted Node MUST not be evicted if the taint is removed before toleration time ends. */ <del> framework.ConformanceIt("removing taint cancels eviction", func() { </del> <ins> framework.ConformanceIt("removing taint cancels eviction [Disruptive]", func() { </ins> podName := "taint-eviction-4" pod := createPodForTaintsTest(true, 2*additionalWaitPerDeleteSeconds, podName, podName, ns) observedDeletions := make(chan string, 100)

Description: In a multi-pods scenario with tolerationSeconds, the pods MUST be evicted as per the toleration time limit. */ <del> framework.ConformanceIt("evicts pods with minTolerationSeconds", func() { </del> <ins> framework.ConformanceIt("evicts pods with minTolerationSeconds [Disruptive]", func() { </ins> podGroup := "taint-eviction-b" observedDeletions := make(chan string, 100) stopCh := make(chan struct{})