Code change:
Many e2e tests today have a tag in their name, which lets us know which sig is responsible for fixing failing or flaky e2e tests. However, not all tests have such a tag. We should: assign tags to all e2e tests that don't yet have them enforce that no new e2e tests can be without a tag in their name EDIT: proposed path forward (pulled out of and ) rearrange e2e tests into sig-owned packages, - e2e tests that run as part of nodee2e - e2e tests that run as part of e2e and nodee2e add that imports sub-packages so e2e binaries can continue to import to pick up test definitions () ensure each sig-owned package has: a package-local () an OWNERS files with sig members and ( follow these conventions: top-level calls are required, nested calls are not allowed remove usage of , , , and in favor of nested calls are ok, nested calls are not enforce these conventions via some combination of: static analysis regex enforcement against names output during a run while we are here, is vestigial and can be removed (ref: ()) EDIT: Everything now conforms to policies defined above will close this out by adding a blocking test and docs to /sig testing /area test for enforcement (FYI /sig architecture /area conformance some of these are conformance tests (FYI /wg reliability for helping determine appropriate sig ownership (FYI ) /sig release since this is relevant to figuring out who should fix release-blocking tests (FYI /priority important-soon /milestone v1.21 I think it's reasonable to aim to accomplish this prior to code freeze for v1.21
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Many e2e tests today have a tag in their name, which lets us know which sig is responsible for fixing failing or flaky e2e tests. assign tags to all e2e tests that don't yet have them enforce that no new e2e tests can be without a tag in their name /sig testing /area test for enforcement (FYI /sig architecture /area conformance some of these are conformance tests (FYI /wg reliability for helping determine appropriate sig ownership (FYI ) /sig release since this is relevant to figuring out who should fix release-blocking tests (FYI /priority important-soon /milestone v1.21 I think it's reasonable to aim to accomplish this prior to code freeze for v1.21 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Assigning an arbitrary lead from each SIG/WG mentioned, please assign to someone more appropriate if you don't have the bandwidth
I can't see anything sig-network related :thinking: there are a lot of NodeConformance things though ,
/assign for visibility - lots of these are node /sig node
/assign (for sig-node)
I've just sent out for autoscaling tests. It seems that most of the remaining tests are node-related. We should try to fix most of these to see if there is anything that is hidden between those.
/unassign
Sent to address rest of the tests. Echoing the review comment from I agree, we can remove prefix once all the tests are tagged. wdyt ? I'll raise a follow up to clean the prefix.
re: /assign (also maybe knows) at one time I thought Openshift was relying on to filter out upstream tests for certain jobs? I don't see that it's consistently applied some I'm assuming we could drop from our tests? (I said as much here: )
We synthetically apply that (if the source package is in ) so can be dropped. Also note in generally I am massively in favor of this change - I forced this to happen in openshift so some of the unassigned upstream tests got blanket assigned to equivalent teams (i.e. any test in that didn't have a sig had to get one synthetically ) so this will remove the need for it. Having owners just clarified everything, and then the ratcheting "you may not add one without" also ensured folks thought even a tiny bit about ownership.
Fleshing out what I suggested in and Can we agree to do the following? rearrange e2e tests into sig-owned packages, - e2e tests that run as part of nodee2e - e2e tests that run as part of e2e and node_e2e ensure each sig-owned package has: a package-local () an OWNERS files with sig members and ( remove usage of , , , and in favor of add that imports sub-packages so e2e binaries can continue to import to pick up test definitions () Enforcing test ownership then looks like: an e2e test must live in a sig-owned package (or subpackage) an e2e test must be defined by that package's PRs to that package will be assigned / auto-labeled to the appropriate SIG changing test ownership is a move from one package to another, requires approval from both the old SIG and new SIG Implications I don't have the foresight to see as blockers, can you think of any? we disallow the concept of e2e packages containing tests owned by multiple sigs some moves may require imports to change or things like helper functions and fixtures to get extracted Folks seemed to like the idea at the time but I'm seeing PRs move in different directions. How can I help get us on the same page?
+1 for that I think that the PR above as an example is an intermediate step (it's not a different direction, rather just one step instead of multiple at once).
There is one question that I have at this point - what do we do with nested Described, e.g.: this one: [there is KubeDescribe nested in SIGDescribe]
I think that having SIGDescribe twice in those cases doesn't seem to be the best option, and those cases seem to be significant percentage of remaining cases...
I don't have strong opinion here. Mostly curious why prefer the non-explicit variant? Is it to optimize for tests migrations?
There is one question that I have at this point - what do we do with nested Described I can't think of a strong reason to ban them, and I think it would be more trouble than it's worth to come up with alternatives for how they're used now. I can't think of a compile-time way to enforce that "[sig-foo]" doesn't show up in their text, so I'm not sure it's worth keeping/enforcing use of wrappers (they definitely make error reporting worse) So something like: replace nested calls with follow these conventions: top-level calls are required, nested calls are not allowed nested calls are ok, nested calls are not enforcement of these conventions can be done via some combination of static analysis and regex check against the full names generated by a run
That makes sense to me. I will start looking into migrating existing tests to satisfy these rules in free cycle, but i'm leaving the enforcement to you :)
I do consider not having to modify test code to change ownership a benefit. But it's more that we'd have to change a lot more code to conform to than . Since we mostly already have the pattern in place, I'd rather stick to it.
Thinking of timeline (ref: ) This work can continue past Code Freeze on Mar 9, but that's a useful milestone to evaluate our progress I'll update the issue description as we get tests owned My goal is to get enforcement landed sometime after Code Freeze This work can't continue past Test Freeze on Mar 24
- also one more thing to do would be to ensure that no tests are tagged with more then one SIG (which I think isn't true now for a few tests)
also one more thing to do would be to ensure that no tests are tagged with more then one SIG (which I think isn't true now for a few tests) Good point. There are tinier corner cases, but let's start with the big one. /sig storage /sig windows /assign There are a bunch of storage tests that look like the following: Which SIG should own these?
We make us of for many e2e tests that are intended to run on windows machines. My suggestion here is probably to have sig-storage own these tests as they reside under /test/e2e/storage/* and switch to using instead of in the test names. SIG-windows will always be happy to help debug issues here. - thought?
I think it makes sense for storage to own the windows storage tests. Using [Feature:Windows] seems reasonable although we'll have to update a number of test jobs.
Working on in-tree enforcement before I update description. tl;dr the majority of tests listed in the description are still not tagged (with at least one SIG), waiting on SIG Node to sign off on usage of KubeDescribe has gone way down, but not at zero yet
is the biggest bit now (once I get feedback from SIG node, we will be almost there with both)
refreshing description based on script in
Regarding "multi-ownership", with windows test figured (the PR didn't merge yet, but hopefully will soon), the biggest question is now about the upgrade tests. - there are a number of apps related upgrade tests. Even though they are currently running on GCP, I suggest that we officially make them sig-apps tests (I'm sure sig-cloud-provider-gcp will be happy to help on infrastructural problems). Thoughts?
can you be more specific about which particular tests? I quickly checked and those are already marked as sig-apps. I don't have any objections as long as they are part of sig-apps territory but I'd like to know a concrete list before saying so :wink:
- sorry, I should have been more specific. Many of those tests are actually being run from here: and the tests there are marked as [sig-cloud-provider-gcp] As part of ensuring that each test has a one single sig ownership, I suggest making it explicit that all of those are sig-apps. Does that make more sense now?
most - yes, but I see a few exceptions: upgradeTest: ServiceUpgradeTest - seems like sig-network SecretUpgradeTest - looks like assigned to sig-storage/sig-apimachinery ConfigMapUpgradeTest - looks like assigned to sig-storage/sig-apimachinery HPAUpgradeTest - sig-autoscaling PersistentVolumeUpgradeTest - sig-storage AppArmorUpgradeTest - sig-auth (?) VolumeModeDowngradeTest - sig-storage gpuUpgradeTests - looks like sig-node/sig-scheduling kubeProxyUpgradeTests - both sig-network kubeProxyDowngradeTests - both sig-network serviceaccountAdmissionControllerMigrationTests - looks like sig-apimachinery the remaining less than half :+1: for sig-apps.
Sure - I didn't mean to put all of them to sig-apps. There is just the biggest group of apps related one so wanted to start with this group.
I'm just making sure :wink: Go for it and thx :100:
Hello! The code freeze is starting March 9th, 2021 (about 1 week from now). As the Issue is tagged for 1.21, is it still planned for this release?
Yes this is planned for this release, and PRs related to this are not subject to code freeze. This will be completed by test freeze (see timeline here: )
- calling your script from head, I have: So there are just couple exceptions now (and a mess with upgrade tests :) )
- when you have time, there are couple PRs waiting for you now
Not reviewing on the weekend, but will check them out first thing next week. Also ok if someone else gets to them before me
Run in master. The is due to The are all from It seems it is done.
The two opened PRs + the one that I have internally (close to ready, though dependent on the firs from the above) should actually close this.
will close this out by adding a blocking test and docs to
Nice work ... that jq stuff is something :)
/triage accepted

Related discussion:
text: <ins> #!/usr/bin/env bash # Copyright 2014 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script verifies the following e2e test ownership policies # - tests MUST start with [sig-foo] # - tests MUST use top-level SIGDescribe # - tests SHOULD NOT have multiple [sig-foo] tags # - tests MUST NOT use nested SIGDescribe # TODO: these two can be dropped if KubeDescribe is gone from codebase # - tests MUST NOT have [k8s.io] in test names # - tests MUST NOT use KubeDescribe set -o errexit set -o nounset set -o pipefail # This will canonicalize the path KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")"/.. && pwd -P) source "${KUBE_ROOT}/hack/lib/init.sh" # Set REUSE_BUILD_OUTPUT=y to skip rebuilding dependencies if present REUSE_BUILD_OUTPUT=${REUSE_BUILD_OUTPUT:-n} # set VERBOSE_OUTPUT=y to output .jq files and shell commands VERBOSE_OUTPUT=${VERBOSE_OUTPUT:-n} if [[ ${VERBOSE_OUTPUT} =~ ^[yY]$ ]]; then set -x fi pushd "${KUBE_ROOT}" > /dev/null # Setup a tmpdir to hold generated scripts and results readonly tmpdir=$(mktemp -d -t verify-e2e-test-ownership.XXXX) trap 'rm -rf ${tmpdir}' EXIT # input spec_summaries="${KUBE_ROOT}/_output/specsummaries.json" # output results_json="${tmpdir}/results.json" summary_json="${tmpdir}/summary.json" failures_json="${tmpdir}/failures.json" # rebuild dependencies if necessary function ensure_dependencies() { local -r ginkgo="${KUBE_ROOT}/_output/bin/ginkgo" local -r e2e_test="${KUBE_ROOT}/_output/bin/e2e.test" if ! { [ -f "${ginkgo}" ] && [[ "${REUSE_BUILD_OUTPUT}" =~ ^[yY]$ ]]; }; then make ginkgo fi if ! { [ -f "${e2e_test}" ] && [[ "${REUSE_BUILD_OUTPUT}" =~ ^[yY]$ ]]; }; then hack/make-rules/build.sh test/e2e/e2e.test fi if ! { [ -f "${spec_summaries}" ] && [[ "${REUSE_BUILD_OUTPUT}" =~ ^[yY]$ ]]; }; then "${ginkgo}" --dryRun=true "${e2e_test}" -- --spec-dump "${spec_summaries}" > /dev/null fi } # evaluate ginkgo spec summaries against e2e test ownership polices # output to ${results_json} function generate_results_json() { readonly results_jq=${tmpdir}/results.jq cat >"${results_jq}" <<EOS [.[] | . as { ComponentTexts: $text, ComponentCodeLocations: $code } | { calls: [ $text | range(1;length) as $i | { sig: (($text[$i] | match("[(sig-[^]]+)]") | .captures[0].string) // "unknown"), text: $text[$i], # unused, but if we ever wanted to have policies based on other tags... # tags: $text[$i] | [match("([[^]]+])"; "g").string], line: $code[$i] | "(.FileName):(.LineNumber)", } + ($code[$i] | .FullStackTrace | match("^(.*).(.+)(.*n") | .captures | { package: .[0].string, func: .[1].string })], } | { owner: .calls[0].sig, testname: .calls | map(.text) | join(" "), calls, policies: [( .calls[0] | { fail: (.sig == "unknown"), level: "FAIL", category: "unowned_test", reason: "must start with [sig-foo]", found: ., }, { fail: (.func != "SIGDescribe"), level: "FAIL", category: "no_sig_describe", reason: "must use top-level SIGDescribe", found: ., } ), ( .calls[1:] | (map(select(.sig != "unknown")) // [] | { fail: . | any, level: "WARN", category: "too_many_sigs", reason: "should not have multiple [sig-foo] tags", found: ., }), (map(select(.func == "SIGDescribe")) // [] | { fail: . | any, level: "FAIL", category: "nested_sig_describe", reason: "must not use nested SIGDescribe", found: ., }) ) ] }] EOS if [[ ${VERBOSE_OUTPUT} =~ ^[yY]$ ]]; then echo "about to  ${results_jq}..." cat -n "${results_jq}" echo fi <"${spec_summaries}" jq --slurp --from-file "${results_jq}" > "${results_json}" } # summarize e2e test policy results # output to ${summary_json} function generate_summary_json() { summary_jq=${tmpdir}/summary.jq cat >"${summary_jq}" <<EOS . as $results | # for each policy category reduce $results[0].policies[] as $p ({}; . + { # add a convenience .policy field containing that policy's result ($p.category): $results | map(. + {policy: .policies[] | select(.category == $p.category)}) | { level: $p.level, reason: $p.reason, passing: map(select(.policy.fail | not)) | length, failing: map(select(.policy.fail)) | length, testnames: map(select(.policy.fail) | .testname), } }) # add a meta policy based on whether any policy failed + { all_policies: $results | { level: "WARN", reason: "should pass all policies", passing: map(select(.policies | map(.fail) | any | not)) | length, failing: map(select(.policies | map(.fail) | any)) | length, testnames: map(select(.policies | map(.fail) | any) | .testname), } } # if a policy has no failing tests, change its log output to PASS | with_entries(.value += { log: (if (.value.failing == 0) then "PASS" else .value.level end) }) # sort by policies with the most failing tests first | to_entries | sort_by(.value.failing) | reverse | from_entries EOS if [[ ${VERBOSE_OUTPUT} =~ ^[yY]$ ]]; then echo "about to run ${results_jq}..." cat -n "${summary_jq}" echo fi <"${results_json}" jq --from-file "${summary_jq}" > "${summary_json}" } # filter e2e policy tests results to tests that failed, with the policies they failed # output to ${failures_json} function generate_failures_json() { local -r failures_jq="${tmpdir}/failures.jq" cat >"${failures_jq}" <<EOS . # for each test | map( # filter down to failing policies; trim category, .reason is more verbose .policies |= map(select(.fail) | del(.category)) # trim the full callstack, .found will contain the relevant call | del(.calls) ) # filter down to tests that have failed policies | map(select(.policies | map (.fail) | any)) EOS if [[ ${VERBOSE_OUTPUT} =~ ^[yY]$ ]]; then echo "about to run ${failures_jq}..." cat -n "${failures_jq}" echo fi <"${results_json}" jq --from-file "${failures_jq}" > "${failures_json}" } function output_results_and_exit_if_failed() { local -r total_tests=$(<"${spec_summaries}" wc -l | awk '{print $1}') # output results to console ( echo "run at datetime: $(date -u +%Y-%m-%dT%H:%M:%SZ)" echo "based on commit: $(git log -n1 --date=iso-strict --pretty='%h - %cd - %s')" echo <"${failures_json}" cat printf "%4s: e2e tests %-40s: %-4dn" "INFO" "in total" "${total_tests}" <"${summary_json}" jq -r 'to_entries[].value | "printf "%4s: ..failing %-40s: %-4dn" "(.log)" "(.reason)" "(.failing)""' | sh ) | tee "${tmpdir}/output.txt" # if we said "FAIL" in that output, we should fail if <"${tmpdir}/output.txt" grep -q "^FAIL"; then echo "FAIL" exit 1 fi } ensure_dependencies generate_results_json generate_failures_json generate_summary_json output_results_and_exit_if_failed echo "PASS" </ins>