Change:
https://k8s- Failed: Kubemark {} Previous issues for this test:
Builds: ci-kubernetes-kubemark-5-gce Failed: Kubemark {}
There are few things here: 3443 run - can you please invstigate it? 3480 run - gcloud copy-files should be retried 3550 run - we should better handle "already exists" errors (and not retry them). I think was looking into it at some point, so he may have already did something
https://k8s- Failed: Kubemark {}
I was looking at 'already exists' problem in RouteController, not in Kubemark.
Builds: ci-kubernetes-kubemark-5-gce Failed: Kubemark {}
Here are the causes of the above tests to fail and fixes: Test Fix ------ ------ ci-kubernetes-kubemark-5-gce/3443 Ignore (as its super flaky) or change create-certs/generate-certs to use a different directory ci-kubernetes-kubemark-5-gce/3480 Run the copy-files with retries as it is idempotent ci-kubernetes-kubemark-5-gce/3550 Don't retry when resource already exists ci-kubernetes-kubemark-5-gce/3666 Not worry as it is flaky or add retrying logic to all gclouds in that script ci-kubernetes-kubemark-5-gce/3787 Run the ssh with retries as it is idempotent
This PR fixes issues with the 2nd, 3rd and 5th tests in the above table. Could you PTAL at the PR? Also, I think we can safely ignore the 1st test (which is super flaky) and what do you suggest about the 4th test (do we add retrying logic to just call inside detect-master function of or let it stay as it is because there are a lot of other such gcloud calls inside the file, all of which should be changed rightfully)?
Let's not fix everything - fix just the things that you need, and direct vicinity.
Ok. So then let's ignore the 1st and the 4th tests in the above table as they are not really important. 2nd, 3rd and 5th are solved by PR . And regarding failure of , it looks like it is a flaky issue as well. The density test succeeded, but the load test failed. From the logs it appears to be an issue with the client caches returning old resource versions for pod watches, with many such log statements: Does this worry you or you'd deem it to be just some weird flake?
It might mean a number of things, but most likely something gets overloaded/congested. It might be hard to debug, as we're seeing this from time to time, but it doesn't look very serious so I think we never investigated it thoroughly.

Discussion:
# arguments: # $@: all stuff that goes after 'gcloud compute ' function run-gcloud-compute-with-retries { <ins> echo "" > /tmp/gcloud_retries </ins> for attempt in $(seq 1 ${RETRIES}); do <del> if ! gcloud compute $@; then </del> <ins> if ! gcloud compute $@ &> /tmp/gcloud_retries; then if [[ $(grep -c "already exists" /tmp/gcloud_retries) -gt 0 ]]; then if [[ "${attempt}" == 1 ]]; then echo -e "${color_red} Failed to $1 $2 $3 as the resource hasn't been deleted from a previous run.${color_norm}" >& 2 exit 1 fi echo -e "${color_yellow}Succeeded to $1 $2 $3 in the previous attempt, but status response wasn't received.${color_norm}" return 0 fi </ins> echo -e "${color_yellow}Attempt $(($attempt+1)) failed to $1 $2 $3. Retrying.${color_norm}" >& 2 sleep $(($attempt * 5)) else

password=$(python -c 'import string,random; print("".join(random.SystemRandom().choice(string.ascii_letters + string.digits) for _ in range(16)))') <del> gcloud compute ssh --zone="${ZONE}" --project="${PROJECT}" "${MASTER_NAME}"  </del> <ins> run-gcloud-compute-with-retries ssh --zone="${ZONE}" --project="${PROJECT}" "${MASTER_NAME}"  </ins> --command="sudo mkdir /home/kubernetes -p && sudo mkdir /etc/srv/kubernetes -p &&  sudo bash -c "echo ${CA_CERT_BASE64} | base64 --decode > /etc/srv/kubernetes/ca.crt" &&  sudo bash -c "echo ${MASTER_CERT_BASE64} | base64 --decode > /etc/srv/kubernetes/server.cert" && 

sudo bash -c "echo "${KUBE_PROXY_TOKEN},system:kube-proxy,uid:kube_proxy" >> /etc/srv/kubernetes/known_tokens.csv" &&  sudo bash -c "echo ${password},admin,admin > /etc/srv/kubernetes/basic_auth.csv"" <del> gcloud compute copy-files --zone="${ZONE}" --project="${PROJECT}"  </del> <ins> run-gcloud-compute-with-retries copy-files --zone="${ZONE}" --project="${PROJECT}"  </ins> "${SERVER_BINARY_TAR}"  "${RESOURCE_DIRECTORY}/kubemark-master-env.sh"  "${RESOURCE_DIRECTORY}/start-kubemark-master.sh"