Change:
fi } <ins> # Reads kube-env metadata from master and extracts value from provided key. # # Assumed vars: #   MASTER_NAME #   ZONE # # Args: # $1 env key to use function get-env-val() { # TODO(mbforbes): Make this more reliable with retries. gcloud compute ssh --zone ${ZONE} ${MASTER_NAME} --command  "curl --fail --silent -H 'Metadata-Flavor: Google'  'http://metadata/computeMetadata/v1/instance/attributes/kube-env'" 2>/dev/null  | grep ${1} | cut -d : -f 2 | cut -d ' -f 2 } # $1 veresion </ins> function upgrade-nodes() { <del> echo "== Upgrading nodes to ${SERVER_BINARY_TAR_URL}. ==" </del> <ins> local version=${1} local sanitized_version=$(echo ${version} | sed s/"."/-/g) echo "== Upgrading nodes to ${version}. ==" </ins> detect-minion-names <del> get-password set-master-htpasswd kube-update-nodes upgrade </del> <ins> # TODO(mbforbes): Refactor setting scope flags. local -a scope_flags=() if (( "${#MINION_SCOPES[@]}" > 0 )); then scope_flags=("--scopes" "${MINION_SCOPES[@]}") else scope_flags=("--no-scopes") fi # Get required node tokens. KUBELET_TOKEN=$(get-env-val "KUBELET_TOKEN") KUBE_PROXY_TOKEN=$(get-env-val "KUBE_PROXY_TOKEN") # TODO(mbforbes): How do we ensure kube-env is written in a ${version}- #                 compatible way? write-node-env # TODO(mbforbes): Get configure-vm script from ${version}. (Must plumb this #                 through all create-node-instance-template implementations). create-node-instance-template ${sanitized_version} # Do the actual upgrade. gcloud preview rolling-updates start  --group "${NODE_INSTANCE_PREFIX}-group"  --max-num-concurrent-instances 1  --max-num-failed-instances 0  --project "${PROJECT}"  --zone "${ZONE}"  --template "${NODE_INSTANCE_PREFIX}-template-${sanitized_version}" </ins> echo "== Done ==" }

Discussion:
See issue as the roll-up for node upgrades. tl;dr: Update MIG template, then call a MIG rollingupdate. More details: When upgrading nodes, an in-place upgrade must worry about leftover state on the boot disk, VM configuration like IP tables, and the idempotency of salt. Getting a fresh VM instead would remove these concerns and provide a clean state for a new node. Furthermore, a versioned node spec () includes the tuple of (kubelet, kube-proxy, docker, kernel/OS). In the general case, changing docker or the kernel/OS means we have to stop the running pods. Once know we have to stop the pods, the simplest upgrade mechanism is to totally down the node and provision a new one. Provisioning a new node is also a great backup to have for node repairs. To be clear, though, this issue is entirely separate from node repairs; once we have the capability to grow the cluster by adding new nodes, this mechanism, and other repair mechanisms, will build on top of that capability. This was blocked by: nodes joining a running cluster (template-only metadata) References: The most relevant discussion regarding the worries of complications with state while doing in-place upgrades is .
FWIW, I think we need this as a fallback to repair completely borked nodes? That is, it may not be necessary for upgrades per se, but could end up being independently useful.
Thanks for the comment, I updated the description to be clear about what this issue is addressing. Having the ability to add new nodes to a cluster will both enable this upgrade path as well as give us a great "hammer" to use for repairs.
Working on this now.