Code change:
Many e2e tests today have a tag in their name, which lets us know which sig is responsible for fixing failing or flaky e2e tests. However, not all tests have such a tag. We should: assign tags to all e2e tests that don't yet have them enforce that no new e2e tests can be without a tag in their name EDIT: proposed path forward (pulled out of and ) rearrange e2e tests into sig-owned packages, - e2e tests that run as part of nodee2e - e2e tests that run as part of e2e and nodee2e add that imports sub-packages so e2e binaries can continue to import to pick up test definitions () ensure each sig-owned package has: a package-local () an OWNERS files with sig members and ( follow these conventions: top-level calls are required, nested calls are not allowed remove usage of , , , and in favor of nested calls are ok, nested calls are not enforce these conventions via some combination of: static analysis regex enforcement against names output during a run while we are here, is vestigial and can be removed (ref: ()) EDIT: Everything now conforms to policies defined above will close this out by adding a blocking test and docs to /sig testing /area test for enforcement (FYI /sig architecture /area conformance some of these are conformance tests (FYI /wg reliability for helping determine appropriate sig ownership (FYI ) /sig release since this is relevant to figuring out who should fix release-blocking tests (FYI /priority important-soon /milestone v1.21 I think it's reasonable to aim to accomplish this prior to code freeze for v1.21
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Many e2e tests today have a tag in their name, which lets us know which sig is responsible for fixing failing or flaky e2e tests. assign tags to all e2e tests that don't yet have them enforce that no new e2e tests can be without a tag in their name /sig testing /area test for enforcement (FYI /sig architecture /area conformance some of these are conformance tests (FYI /wg reliability for helping determine appropriate sig ownership (FYI ) /sig release since this is relevant to figuring out who should fix release-blocking tests (FYI /priority important-soon /milestone v1.21 I think it's reasonable to aim to accomplish this prior to code freeze for v1.21 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Assigning an arbitrary lead from each SIG/WG mentioned, please assign to someone more appropriate if you don't have the bandwidth
I can't see anything sig-network related :thinking: there are a lot of NodeConformance things though ,
/assign for visibility - lots of these are node /sig node
/assign (for sig-node)
I've just sent out for autoscaling tests. It seems that most of the remaining tests are node-related. We should try to fix most of these to see if there is anything that is hidden between those.
/unassign
Sent to address rest of the tests. Echoing the review comment from I agree, we can remove prefix once all the tests are tagged. wdyt ? I'll raise a follow up to clean the prefix.
re: /assign (also maybe knows) at one time I thought Openshift was relying on to filter out upstream tests for certain jobs? I don't see that it's consistently applied some I'm assuming we could drop from our tests? (I said as much here: )
We synthetically apply that (if the source package is in ) so can be dropped. Also note in generally I am massively in favor of this change - I forced this to happen in openshift so some of the unassigned upstream tests got blanket assigned to equivalent teams (i.e. any test in that didn't have a sig had to get one synthetically ) so this will remove the need for it. Having owners just clarified everything, and then the ratcheting "you may not add one without" also ensured folks thought even a tiny bit about ownership.
Fleshing out what I suggested in and Can we agree to do the following? rearrange e2e tests into sig-owned packages, - e2e tests that run as part of nodee2e - e2e tests that run as part of e2e and node_e2e ensure each sig-owned package has: a package-local () an OWNERS files with sig members and ( remove usage of , , , and in favor of add that imports sub-packages so e2e binaries can continue to import to pick up test definitions () Enforcing test ownership then looks like: an e2e test must live in a sig-owned package (or subpackage) an e2e test must be defined by that package's PRs to that package will be assigned / auto-labeled to the appropriate SIG changing test ownership is a move from one package to another, requires approval from both the old SIG and new SIG Implications I don't have the foresight to see as blockers, can you think of any? we disallow the concept of e2e packages containing tests owned by multiple sigs some moves may require imports to change or things like helper functions and fixtures to get extracted Folks seemed to like the idea at the time but I'm seeing PRs move in different directions. How can I help get us on the same page?
+1 for that I think that the PR above as an example is an intermediate step (it's not a different direction, rather just one step instead of multiple at once).
There is one question that I have at this point - what do we do with nested Described, e.g.: this one: [there is KubeDescribe nested in SIGDescribe]
I think that having SIGDescribe twice in those cases doesn't seem to be the best option, and those cases seem to be significant percentage of remaining cases...
I don't have strong opinion here. Mostly curious why prefer the non-explicit variant? Is it to optimize for tests migrations?
There is one question that I have at this point - what do we do with nested Described I can't think of a strong reason to ban them, and I think it would be more trouble than it's worth to come up with alternatives for how they're used now. I can't think of a compile-time way to enforce that "[sig-foo]" doesn't show up in their text, so I'm not sure it's worth keeping/enforcing use of wrappers (they definitely make error reporting worse) So something like: replace nested calls with follow these conventions: top-level calls are required, nested calls are not allowed nested calls are ok, nested calls are not enforcement of these conventions can be done via some combination of static analysis and regex check against the full names generated by a run
That makes sense to me. I will start looking into migrating existing tests to satisfy these rules in free cycle, but i'm leaving the enforcement to you :)
I do consider not having to modify test code to change ownership a benefit. But it's more that we'd have to change a lot more code to conform to than . Since we mostly already have the pattern in place, I'd rather stick to it.
Thinking of timeline (ref: ) This work can continue past Code Freeze on Mar 9, but that's a useful milestone to evaluate our progress I'll update the issue description as we get tests owned My goal is to get enforcement landed sometime after Code Freeze This work can't continue past Test Freeze on Mar 24
- also one more thing to do would be to ensure that no tests are tagged with more then one SIG (which I think isn't true now for a few tests)
also one more thing to do would be to ensure that no tests are tagged with more then one SIG (which I think isn't true now for a few tests) Good point. There are tinier corner cases, but let's start with the big one. /sig storage /sig windows /assign There are a bunch of storage tests that look like the following: Which SIG should own these?
We make us of for many e2e tests that are intended to run on windows machines. My suggestion here is probably to have sig-storage own these tests as they reside under /test/e2e/storage/* and switch to using instead of in the test names. SIG-windows will always be happy to help debug issues here. - thought?
I think it makes sense for storage to own the windows storage tests. Using [Feature:Windows] seems reasonable although we'll have to update a number of test jobs.
Working on in-tree enforcement before I update description. tl;dr the majority of tests listed in the description are still not tagged (with at least one SIG), waiting on SIG Node to sign off on usage of KubeDescribe has gone way down, but not at zero yet
is the biggest bit now (once I get feedback from SIG node, we will be almost there with both)
refreshing description based on script in
Regarding "multi-ownership", with windows test figured (the PR didn't merge yet, but hopefully will soon), the biggest question is now about the upgrade tests. - there are a number of apps related upgrade tests. Even though they are currently running on GCP, I suggest that we officially make them sig-apps tests (I'm sure sig-cloud-provider-gcp will be happy to help on infrastructural problems). Thoughts?
can you be more specific about which particular tests? I quickly checked and those are already marked as sig-apps. I don't have any objections as long as they are part of sig-apps territory but I'd like to know a concrete list before saying so :wink:
- sorry, I should have been more specific. Many of those tests are actually being run from here: and the tests there are marked as [sig-cloud-provider-gcp] As part of ensuring that each test has a one single sig ownership, I suggest making it explicit that all of those are sig-apps. Does that make more sense now?
most - yes, but I see a few exceptions: upgradeTest: ServiceUpgradeTest - seems like sig-network SecretUpgradeTest - looks like assigned to sig-storage/sig-apimachinery ConfigMapUpgradeTest - looks like assigned to sig-storage/sig-apimachinery HPAUpgradeTest - sig-autoscaling PersistentVolumeUpgradeTest - sig-storage AppArmorUpgradeTest - sig-auth (?) VolumeModeDowngradeTest - sig-storage gpuUpgradeTests - looks like sig-node/sig-scheduling kubeProxyUpgradeTests - both sig-network kubeProxyDowngradeTests - both sig-network serviceaccountAdmissionControllerMigrationTests - looks like sig-apimachinery the remaining less than half :+1: for sig-apps.
Sure - I didn't mean to put all of them to sig-apps. There is just the biggest group of apps related one so wanted to start with this group.
I'm just making sure :wink: Go for it and thx :100:
Hello! The code freeze is starting March 9th, 2021 (about 1 week from now). As the Issue is tagged for 1.21, is it still planned for this release?
Yes this is planned for this release, and PRs related to this are not subject to code freeze. This will be completed by test freeze (see timeline here: )
- calling your script from head, I have: So there are just couple exceptions now (and a mess with upgrade tests :) )
- when you have time, there are couple PRs waiting for you now
Not reviewing on the weekend, but will check them out first thing next week. Also ok if someone else gets to them before me
Run in master. The is due to The are all from It seems it is done.
The two opened PRs + the one that I have internally (close to ready, though dependent on the firs from the above) should actually close this.
will close this out by adding a blocking test and docs to
Nice work ... that jq stuff is something :)
/triage accepted

Related discussion:
text: <del> See [e2e-tests](https://git.k8s.io/community/contributors/devel/sig-testing/e2e-tests.md) </del> <ins> # test/e2e This is home to e2e tests used for presubmit, periodic, and postsubmit jobs. Some of these jobs are merge-blocking, some are release-blocking. ## e2e test ownership All e2e tests must adhere to the following policies: - the test must be owned by one and only one SIG - the test must live in/underneath a sig-owned package matching pattern: `test/e2e/[{subpath}/]{sig}/...`, e.g. - `test/e2e/auth` - all tests owned by sig-`auth` - `test/e2e/common/storage` - all tests `common` to cluster-level and node-level e2e tests, owned by sig-`node` - `test/e2e/upgrade/apps` - all tests used in `upgrade` testing, owned by sig-`apps` - each sig-owned package should have an OWNERS file defining relevant approvers and labels for the owning sig, e.g. ```yaml # test/e2e/node/OWNERS # See the OWNERS docs at https://go.k8s.io/owners approvers: - alice - bob - cynthia emeritus_approvers: - dave reviewers: - sig-node-reviewers labels: - sig/node ``` - packages that use `{subpath}` should have an `imports.go` file importing sig-owned packages (for ginkgo's benefit), e.g. ```golang // test/e2e/common/imports.go package common import ( // ensure these packages are scanned by ginkgo for e2e tests _ "k8s.io/kubernetes/test/e2e/common/network" _ "k8s.io/kubernetes/test/e2e/common/node" _ "k8s.io/kubernetes/test/e2e/common/storage" ) ``` - test ownership must be declared via a top-level SIGDescribe call defined in the sig-owned package, e.g. ```golang // test/e2e/lifecycle/framework.go package lifecycle import "github.com/onsi/ginkgo" // SIGDescribe annotates the test with the SIG label. func SIGDescribe(text string, body func()) bool { return ginkgo.Describe("[sig-cluster-lifecycle] "+text, body) } ``` ```golang // test/e2e/lifecycle/bootstrap/bootstrap_signer.go package bootstrap import ( "github.com/onsi/ginkgo" "k8s.io/kubernetes/test/e2e/lifecycle" ) var _ = lifecycle.SIGDescribe("[Feature:BootstrapTokens]", func() { /* ... */ ginkgo.It("should sign the new added bootstrap tokens", func() { /* ... */ }) /* etc */ }) ``` These polices are enforced: - via the merge-blocking presubmit job `pull-kubernetes-verify` - which ends up running `hack/verify-e2e-test-ownership.sh` - which can also be run via `make verify WHAT=e2e-test-ownership` ## more info See [kubernetes/community/.../e2e-tests.md](https://git.k8s.io/community/contributors/devel/sig-testing/e2e-tests.md) </ins>