Change:
kube::golang::setup_env API_REFERENCE_DOCS_ROOT="${KUBE_ROOT}/docs/api-reference" <ins> # Use REPO_DIR if provided so we can set it to the host-resolvable path # to the repo root if we are running this script from a container with # docker mounted in as a volume. # We pass the host output dir to update-api-reference-docs.sh, but use # the regular one to compute diff (they will be the same if running this # test on the host, potentially different if running in a container). REPO_DIR=${REPO_DIR:-"${KUBE_ROOT}"} HOST_OUTPUT_DIR="${REPO_DIR}/_tmp/api-reference" </ins> TMP_OUTPUT_DIR="${KUBE_ROOT}/_tmp/api-reference" TMP_ROOT="${KUBE_ROOT}/_tmp" # Generate API reference docs in tmp. <del> "./hack/update-api-reference-docs.sh" "${TMP_OUTPUT_DIR}" </del> <ins> "./hack/update-api-reference-docs.sh" "${HOST_OUTPUT_DIR}" </ins> echo "diffing ${API_REFERENCE_DOCS_ROOT} against freshly generated docs" ret=0

Discussion:
In the pull request Jenkins, we're running up to 8 instances of verification checks, build, unit tests, integration tests, and e2e tests. Unfortunately, it seems like several of these components were not designed to run simultaneous to other tests on the same machine. In particular, several unit tests seem to write to static file names in /tmp (or equivalent), which is a fairly easy fix. Some of the verification checks and the integration test start up etcd, however, and the etcd helper checks that only a single etcd binary is running on the host (and uses a static port for it). For example: We might be able to fix the etcd issue separately, or we can just figure out how to containerize all of these tests so they can't possibly interfere with each other. Note that we're ignoring the exit status of the verification checks and unit/integration tests right now, so these etcd failures are effectively being ignored (though it is causing us to miss test coverage).
What percentage of tests are failing due to being non-hermetic like this?
I'm not sure about unit tests (I've anecdotally seen just a few fail), but the verification checks are failing fairly regularly, and the integration test is also similarly frequently not running due to this.
To be clear, I'm looking for real numbers, to decide whether it's feasible to temporarily disable a small number of tests until we can hermetify them. Otherwise we need to turn off all unit, integration and verification tests in Jenkins to avoid irritating invalid regression signals. We need to do one or the other this week.
I'll try to dig through the logs, figure out which ones have failed recently, disable them in Jenkins, and see whether the flakes abate.
I just turned off all unit, integration and verification checks in Jenkins, as a temporary measure to unstall the merge-bot.
As a point of information, I just found approximately 20 tests in which use a static tmp path with semi-random filenames which are highly likely to collide. We could keep searching for unit tests that are non-hermetic, but it'd probably be a better investment of time to fix the test infrastructure so test authors don't need to care (and so we don't need to go look at and fix ~2000 test cases).

Another few metrics, using the string "/tmp" as a likely anti-pattern: We'd need to update each of these 23 files to get them to use or if they aren't already.
Created to track cleaning up tempdir usage in unit tests.
cc