Change:
'kubernetes-soak-weekly-deploy' sometimes fails to deploy because the clean-up cluster step fails to delete the template: Then the subsequent cluster step fails to create the template, because it already exists: It keeps retrying and failing, because the template already exists, until it times out and the deployment fails. Suggested fixes: Cluster down should try harder to delete the template (maybe wait/retry). Cluster up should recognize that the template exists, and try to delete/recreate, or just re-use the existing.
Looks like ran in to this on Monday, May 11, 2015 around 1:00 PM PST. I'll assign this to my self and take a crack at fixing it.
This is causing most of our e2e tests to fail in continuous integration as well as users private e2e test clusters. Bumping to priority/p0 and v1.0 milestone. Saad thinks he's figured it out, and is working on a fix as we speak.
In essence... Saad: "I think what's happening is the previous clean up step to delete the managed instance group happens asynchronously, therefore there is a race condition to delete the template. Working on a fix".

Discussion:
detect-project echo "Bringing down cluster" <ins> set +e  # Do not stop on error </ins> <del> gcloud preview managed-instance-groups --zone "${ZONE}" delete  </del> <ins> # The gcloud APIs don't return machine parsable error codes/retry information. Therefore the best we can # do is parse the output and special case particular responses we are interested in. deleteCmdOutput=$(gcloud preview managed-instance-groups --zone "${ZONE}" delete  </ins> --project "${PROJECT}"  --quiet  <del> "${NODE_INSTANCE_PREFIX}-group" || true </del> <ins> "${NODE_INSTANCE_PREFIX}-group") if [[ "$deleteCmdOutput" != ""  ]]; then # Managed instance group deletion is done asyncronously, we must wait for it to complete, or subsequent steps fail deleteCmdOperationId=$(echo $deleteCmdOutput | grep "Operation:" | sed "s/.*Operation:s//" | sed "s/s.*//" | sed "s/ //g") if [[ "$deleteCmdOperationId" != ""  ]]; then deleteCmdStatus="PENDING" while [[ "$deleteCmdStatus" != "DONE" ]] do sleep 5 deleteCmdOperationOutput=$(gcloud preview managed-instance-groups --zone "${ZONE}" get-operation $deleteCmdOperationId) deleteCmdStatus=$(echo $deleteCmdOperationOutput | grep -i "status:" | sed "s/.*status:s//" | sed "s/s.*//" | sed "s/ //g") echo "Waiting for MIG deletion to complete. Current status: " $deleteCmdStatus done fi fi </ins> gcloud compute instance-templates delete  --project "${PROJECT}"  --quiet  <del> "${NODE_INSTANCE_PREFIX}-template" || true </del> <ins> "${NODE_INSTANCE_PREFIX}-template" </ins> # First delete the master (if it exists). gcloud compute instances delete 

--quiet  --delete-disks all  --zone "${ZONE}"  <del> "${MASTER_NAME}" || true </del> <ins> "${MASTER_NAME}" </ins> # Delete the master pd (possibly leaked by kube-up if master create failed) gcloud compute disks delete  --project "${PROJECT}"  --quiet  --zone "${ZONE}"  <del> "${MASTER_NAME}"-pd || true </del> <ins> "${MASTER_NAME}"-pd </ins> # Find out what minions are running. local -a minions

--quiet  --delete-disks boot  --zone "${ZONE}"  <del> "${minions[@]::10}" || true </del> <ins> "${minions[@]::10}" </ins> minions=( "${minions[@]:10}" ) done

gcloud compute firewall-rules delete   --project "${PROJECT}"  --quiet  <del> "${MASTER_NAME}-https" || true </del> <ins> "${MASTER_NAME}-https" </ins> # Delete firewall rule for minions. gcloud compute firewall-rules delete   --project "${PROJECT}"  --quiet  <del> "${MINION_TAG}-all" || true </del> <ins> "${MINION_TAG}-all" </ins> # Delete routes. local -a routes

gcloud compute routes delete  --project "${PROJECT}"  --quiet  <del> "${routes[@]::10}" || true </del> <ins> "${routes[@]::10}" </ins> routes=( "${routes[@]:10}" ) done

--project "${PROJECT}"  --region "${REGION}"  --quiet  <del> "${MASTER_NAME}-ip" || true </del> <ins> "${MASTER_NAME}-ip" </ins> export CONTEXT="${PROJECT}_${INSTANCE_PREFIX}" clear-kubeconfig <ins> set -e </ins> } # Update a kubernetes cluster with latest source