Change:
<ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail set -o xtrace # Runs the unit and integration tests, producing JUnit-style XML test # reports in ${WORKSPACE}/artifacts. This script is intended to be run from # kubekins-test container with a kubernetes repo mapped in. See # hack/jenkins/gotest-dockerized.sh export PATH=${GOPATH}/bin:${PWD}/third_party/etcd:/usr/local/go/bin:${PATH} go get github.com/tools/godep go get github.com/jstemmer/go-junit-report # Enable the Go race detector. export KUBE_RACE=-race # Disable coverage report export KUBE_COVER="n" # Produce a JUnit-style XML test report for Jenkins. export KUBE_JUNIT_REPORT_DIR=${WORKSPACE}/artifacts # Save the verbose stdout as well. export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y export KUBE_TIMEOUT='-timeout 300s' export KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=4 export LOG_LEVEL=4 ./hack/build-go.sh godep go install ./... ./hack/install-etcd.sh ./hack/verify-all.sh ./hack/test-go.sh -- -p=2 ./hack/test-cmd.sh ./hack/test-integration.sh ./hack/test-update-storage-objects.sh </ins>

Discussion:
In the pull request Jenkins, we're running up to 8 instances of verification checks, build, unit tests, integration tests, and e2e tests. Unfortunately, it seems like several of these components were not designed to run simultaneous to other tests on the same machine. In particular, several unit tests seem to write to static file names in /tmp (or equivalent), which is a fairly easy fix. Some of the verification checks and the integration test start up etcd, however, and the etcd helper checks that only a single etcd binary is running on the host (and uses a static port for it). For example: We might be able to fix the etcd issue separately, or we can just figure out how to containerize all of these tests so they can't possibly interfere with each other. Note that we're ignoring the exit status of the verification checks and unit/integration tests right now, so these etcd failures are effectively being ignored (though it is causing us to miss test coverage).
What percentage of tests are failing due to being non-hermetic like this?
I'm not sure about unit tests (I've anecdotally seen just a few fail), but the verification checks are failing fairly regularly, and the integration test is also similarly frequently not running due to this.
To be clear, I'm looking for real numbers, to decide whether it's feasible to temporarily disable a small number of tests until we can hermetify them. Otherwise we need to turn off all unit, integration and verification tests in Jenkins to avoid irritating invalid regression signals. We need to do one or the other this week.
I'll try to dig through the logs, figure out which ones have failed recently, disable them in Jenkins, and see whether the flakes abate.
I just turned off all unit, integration and verification checks in Jenkins, as a temporary measure to unstall the merge-bot.
As a point of information, I just found approximately 20 tests in which use a static tmp path with semi-random filenames which are highly likely to collide. We could keep searching for unit tests that are non-hermetic, but it'd probably be a better investment of time to fix the test infrastructure so test authors don't need to care (and so we don't need to go look at and fix ~2000 test cases).

Another few metrics, using the string "/tmp" as a likely anti-pattern: We'd need to update each of these 23 files to get them to use or if they aren't already.
Created to track cleaning up tempdir usage in unit tests.
cc