Change:
In the pull request Jenkins, we're running up to 8 instances of verification checks, build, unit tests, integration tests, and e2e tests. Unfortunately, it seems like several of these components were not designed to run simultaneous to other tests on the same machine. In particular, several unit tests seem to write to static file names in /tmp (or equivalent), which is a fairly easy fix. Some of the verification checks and the integration test start up etcd, however, and the etcd helper checks that only a single etcd binary is running on the host (and uses a static port for it). For example: We might be able to fix the etcd issue separately, or we can just figure out how to containerize all of these tests so they can't possibly interfere with each other. Note that we're ignoring the exit status of the verification checks and unit/integration tests right now, so these etcd failures are effectively being ignored (though it is causing us to miss test coverage).
What percentage of tests are failing due to being non-hermetic like this?
I'm not sure about unit tests (I've anecdotally seen just a few fail), but the verification checks are failing fairly regularly, and the integration test is also similarly frequently not running due to this.
To be clear, I'm looking for real numbers, to decide whether it's feasible to temporarily disable a small number of tests until we can hermetify them. Otherwise we need to turn off all unit, integration and verification tests in Jenkins to avoid irritating invalid regression signals. We need to do one or the other this week.
I'll try to dig through the logs, figure out which ones have failed recently, disable them in Jenkins, and see whether the flakes abate.
I just turned off all unit, integration and verification checks in Jenkins, as a temporary measure to unstall the merge-bot.
As a point of information, I just found approximately 20 tests in which use a static tmp path with semi-random filenames which are highly likely to collide. We could keep searching for unit tests that are non-hermetic, but it'd probably be a better investment of time to fix the test infrastructure so test authors don't need to care (and so we don't need to go look at and fix ~2000 test cases).
Another few metrics, using the string "/tmp" as a likely anti-pattern: We'd need to update each of these 23 files to get them to use or if they aren't already.
Created to track cleaning up tempdir usage in unit tests.
cc

Discussion:
<ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail set -o xtrace export REPO_DIR=${REPO_DIR:-$(pwd)} # Produce a JUnit-style XML test report for Jenkins. export KUBE_JUNIT_REPORT_DIR=${WORKSPACE}/_artifacts # Run the kubekins container, mapping in docker (so we can launch containers), # the repo directory, and the artifacts output directory. # # Note: We pass in the absolute path to the repo on the host as an env var incase # any tests that get run need to launch containers that also map volumes. # This is required because if you do # # $ docker run -v $PATH:/container/path ... # # From _inside_ a container that has the host's docker mapped in, the $PATH # provided must be resolvable on the *HOST*, not the container. docker run --rm=true  -v /var/run/docker.sock:/var/run/docker.sock  -v "$(which docker)":/bin/docker  -v "${REPO_DIR}":/go/src/k8s.io/kubernetes  -v "${KUBE_JUNIT_REPORT_DIR}":/workspace/artifacts  --env REPO_DIR="${REPO_DIR}"  -it gcr.io/google_containers/kubekins-test:0.1  bash -c "cd kubernetes && ./hack/jenkins/test-dockerized.sh" </ins>

<del> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail set -o xtrace export REPO_DIR=${REPO_DIR:-$(pwd)} # Produce a JUnit-style XML test report for Jenkins. export KUBE_JUNIT_REPORT_DIR=${WORKSPACE}/_artifacts # Run the kubekins container, mapping in docker (so we can launch containers) # and the repo directory docker run -v /var/run/docker.sock:/var/run/docker.sock  -v "$(which docker)":/bin/docker  -v "${REPO_DIR}":/go/src/k8s.io/kubernetes  -v "${KUBE_JUNIT_REPORT_DIR}":/workspace/artifacts  -it kubekins-test  bash -c "cd kubernetes && /workspace/run.sh" </del>

<ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail set -o xtrace # Runs the unit and integration tests, producing JUnit-style XML test # reports in ${WORKSPACE}/artifacts. This script is intended to be run from # kubekins-test container with a kubernetes repo mapped in. See # hack/jenkins/gotest-dockerized.sh export PATH=${GOPATH}/bin:${PWD}/third_party/etcd:/usr/local/go/bin:${PATH} go get github.com/tools/godep go get github.com/jstemmer/go-junit-report # Enable the Go race detector. export KUBE_RACE=-race # Disable coverage report export KUBE_COVER="n" # Produce a JUnit-style XML test report for Jenkins. export KUBE_JUNIT_REPORT_DIR=${WORKSPACE}/artifacts # Save the verbose stdout as well. export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y export KUBE_TIMEOUT='-timeout 300s' export KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=4 export LOG_LEVEL=4 ./hack/build-go.sh godep go install ./... ./hack/install-etcd.sh ./hack/verify-all.sh ./hack/test-go.sh -- -p=2 ./hack/test-cmd.sh ./hack/test-integration.sh ./hack/test-update-storage-objects.sh </ins>

# unit and integration tests FROM golang:1.4 <del> MAINTAINER  jeff lowdermilk <jeffml@google.com> </del> <ins> MAINTAINER  Jeff Lowdermilk <jeffml@google.com> </ins> ENV KUBE_TEST_API_VERSIONS  v1,extensions/v1beta1 ENV KUBE_TEST_ETCD_PREFIXES registry ENV WORKSPACE               /workspace <ins> ENV TERM                    xterm </ins> WORKDIR /workspace <del> ADD run.sh ./ </del> RUN apt-get -o Acquire::Check-Valid-Until=false update && apt-get install -y rsync <ins> RUN apt-get install -y file </ins> RUN mkdir -p /go/src/k8s.io/kubernetes RUN ln -s /go/src/k8s.io/kubernetes /workspace/kubernetes

<ins> all: push TAG = 0.1 container: docker build -t gcr.io/google_containers/kubekins-test . docker tag gcr.io/google_containers/kubekins-test gcr.io/google_containers/kubekins-test:$(TAG) push: container gcloud docker push gcr.io/google_containers/kubekins-test # Push image tagged as latest to repository gcloud docker push gcr.io/google_containers/kubekins-test:$(TAG) # Push version tagged image to repository (since this image is already pushed it will simply create or update version tag) clean: </ins>

<del> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail set -o xtrace export PATH=${GOPATH}/bin:${PWD}/third_party/etcd:/usr/local/go/bin:${PATH} ./hack/install-etcd.sh go get -u github.com/jstemmer/go-junit-report go get golang.org/x/tools/cmd/cover go get github.com/mattn/goveralls go get github.com/tools/godep go get github.com/jstemmer/go-junit-report # Enable the Go race detector. export KUBE_RACE=-race # Disable coverage report export KUBE_COVER="n" # Produce a JUnit-style XML test report for Jenkins. export KUBE_JUNIT_REPORT_DIR=${WORKSPACE}/artifacts # Save the verbose stdout as well. export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y export KUBE_GOVERALLS_BIN="${GOPATH}/bin/goveralls" export KUBE_TIMEOUT='-timeout 300s' export KUBE_COVERPROCS=8 export KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=4 export LOG_LEVEL=4 ./hack/verify-gofmt.sh ./hack/verify-boilerplate.sh ./hack/verify-description.sh ./hack/verify-flags-underscore.py ./hack/verify-generated-conversions.sh ./hack/verify-generated-deep-copies.sh ./hack/verify-generated-docs.sh ./hack/verify-generated-swagger-docs.sh ./hack/verify-swagger-spec.sh ./hack/verify-linkcheck.sh ./hack/test-go.sh -- -p=2 ./hack/test-cmd.sh ./hack/test-integration.sh ./hack/test-update-storage-objects.sh </del>

set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. <del> DEFAULT_OUTPUT_PATH="$PWD/${KUBE_ROOT}/docs/api-reference" </del> <ins> REPO_DIR=${REPO_DIR:-"${PWD}/${KUBE_ROOT}"} DEFAULT_OUTPUT_PATH="${REPO_DIR}/docs/api-reference" </ins> OUTPUT=${1:-${DEFAULT_OUTPUT_PATH}} echo "Generating api reference docs at ${OUTPUT}" V1_PATH="${OUTPUT}/v1/" V1BETA1_PATH="${OUTPUT}/extensions/v1beta1" <del> SWAGGER_PATH="$PWD/${KUBE_ROOT}/api/swagger-spec/" </del> <ins> SWAGGER_PATH="${REPO_DIR}/api/swagger-spec/" </ins> echo "Reading swagger spec from: ${SWAGGER_PATH}"

kube::golang::setup_env API_REFERENCE_DOCS_ROOT="${KUBE_ROOT}/docs/api-reference" <ins> # Use REPO_DIR if provided so we can set it to the host-resolvable path # to the repo root if we are running this script from a container with # docker mounted in as a volume. # We pass the host output dir to update-api-reference-docs.sh, but use # the regular one to compute diff (they will be the same if running this # test on the host, potentially different if running in a container). REPO_DIR=${REPO_DIR:-"${KUBE_ROOT}"} HOST_OUTPUT_DIR="${REPO_DIR}/_tmp/api-reference" </ins> TMP_OUTPUT_DIR="${KUBE_ROOT}/_tmp/api-reference" TMP_ROOT="${KUBE_ROOT}/_tmp" # Generate API reference docs in tmp. <del> "./hack/update-api-reference-docs.sh" "${TMP_OUTPUT_DIR}" </del> <ins> "./hack/update-api-reference-docs.sh" "${HOST_OUTPUT_DIR}" </ins> echo "diffing ${API_REFERENCE_DOCS_ROOT} against freshly generated docs" ret=0