Change:
See issue as the roll-up for node upgrades. tl;dr: Update MIG template, then call a MIG rollingupdate. More details: When upgrading nodes, an in-place upgrade must worry about leftover state on the boot disk, VM configuration like IP tables, and the idempotency of salt. Getting a fresh VM instead would remove these concerns and provide a clean state for a new node. Furthermore, a versioned node spec () includes the tuple of (kubelet, kube-proxy, docker, kernel/OS). In the general case, changing docker or the kernel/OS means we have to stop the running pods. Once know we have to stop the pods, the simplest upgrade mechanism is to totally down the node and provision a new one. Provisioning a new node is also a great backup to have for node repairs. To be clear, though, this issue is entirely separate from node repairs; once we have the capability to grow the cluster by adding new nodes, this mechanism, and other repair mechanisms, will build on top of that capability. This was blocked by: nodes joining a running cluster (template-only metadata) References: The most relevant discussion regarding the worries of complications with state while doing in-place upgrades is .
FWIW, I think we need this as a fallback to repair completely borked nodes? That is, it may not be necessary for upgrades per se, but could end up being independently useful.
Thanks for the comment, I updated the description to be clear about what this issue is addressing. Having the ability to add new nodes to a cluster will both enable this upgrade path as well as give us a great "hammer" to use for repairs.
Working on this now.

Discussion:
# TODO(dawnchen): Check $CONTAINER_RUNTIME to decide which # cloud_config yaml file should be passed <ins> # TODO(mbforbes): Make $1 required. # TODO(mbforbes): Document required vars (for this and call chain). # $1 version </ins> function create-node-instance-template { <del> create-node-template "${NODE_INSTANCE_PREFIX}-template" "${scope_flags[*]}"  </del> <ins> local suffix="" if [[ -n ${1:-} ]]; then suffix="-${1}" fi create-node-template "${NODE_INSTANCE_PREFIX}-template${suffix}" "${scope_flags[*]}"  </ins> "kube-env=${KUBE_TEMP}/node-kube-env.yaml"  "user-data=${KUBE_ROOT}/cluster/gce/coreos/node.yaml" }

--disk name="${MASTER_NAME}-pd" device-name=master-pd mode=rw boot=no auto-delete=no } <ins> # TODO(mbforbes): Make $1 required. # TODO(mbforbes): Document required vars (for this and call chain). # $1 version </ins> function create-node-instance-template { <del> create-node-template "${NODE_INSTANCE_PREFIX}-template" "${scope_flags[*]}"  </del> <ins> local suffix="" if [[ -n ${1:-} ]]; then suffix="-${1}" fi create-node-template "${NODE_INSTANCE_PREFIX}-template${suffix}" "${scope_flags[*]}"  </ins> "startup-script=${KUBE_ROOT}/cluster/gce/configure-vm.sh"  "kube-env=${KUBE_TEMP}/node-kube-env.yaml" }

function upgrade-master() { echo "== Upgrading master to '${SERVER_BINARY_TAR_URL}'. Do not interrupt, deleting master instance. ==" <del> detect-master </del> get-password <del> set-master-htpasswd </del> <ins> get-bearer-token detect-master </ins> # Delete the master instance. Note that the master-pd is created # with auto-delete=no, so it should not be deleted.

fi } <ins> # Reads kube-env metadata from master and extracts value from provided key. # # Assumed vars: #   MASTER_NAME #   ZONE # # Args: # $1 env key to use function get-env-val() { # TODO(mbforbes): Make this more reliable with retries. gcloud compute ssh --zone ${ZONE} ${MASTER_NAME} --command  "curl --fail --silent -H 'Metadata-Flavor: Google'  'http://metadata/computeMetadata/v1/instance/attributes/kube-env'" 2>/dev/null  | grep ${1} | cut -d : -f 2 | cut -d ' -f 2 } # $1 veresion </ins> function upgrade-nodes() { <del> echo "== Upgrading nodes to ${SERVER_BINARY_TAR_URL}. ==" </del> <ins> local version=${1} local sanitized_version=$(echo ${version} | sed s/"."/-/g) echo "== Upgrading nodes to ${version}. ==" </ins> detect-minion-names <del> get-password set-master-htpasswd kube-update-nodes upgrade </del> <ins> # TODO(mbforbes): Refactor setting scope flags. local -a scope_flags=() if (( "${#MINION_SCOPES[@]}" > 0 )); then scope_flags=("--scopes" "${MINION_SCOPES[@]}") else scope_flags=("--no-scopes") fi # Get required node tokens. KUBELET_TOKEN=$(get-env-val "KUBELET_TOKEN") KUBE_PROXY_TOKEN=$(get-env-val "KUBE_PROXY_TOKEN") # TODO(mbforbes): How do we ensure kube-env is written in a ${version}- #                 compatible way? write-node-env # TODO(mbforbes): Get configure-vm script from ${version}. (Must plumb this #                 through all create-node-instance-template implementations). create-node-instance-template ${sanitized_version} # Do the actual upgrade. gcloud preview rolling-updates start  --group "${NODE_INSTANCE_PREFIX}-group"  --max-num-concurrent-instances 1  --max-num-failed-instances 0  --project "${PROJECT}"  --zone "${ZONE}"  --template "${NODE_INSTANCE_PREFIX}-template-${sanitized_version}" </ins> echo "== Done ==" }

fi if [[ "${node_upgrade}" == "true" ]]; then <del> upgrade-nodes </del> <ins> if [[ "${local_binaries}" == "true" ]]; then echo "Upgrading nodes to local binaries is not yet supported." >&2 else upgrade-nodes ${binary_version} fi </ins> fi <ins> echo "== Validating cluster post-upgrade ==" </ins> "${KUBE_ROOT}/cluster/validate-cluster.sh"

# $4: The kube-env metadata. function create-node-template { detect-project <ins> # First, ensure the template doesn't exist. # TODO(mbforbes): To make this really robust, we need to parse the output and #                 add retries. Just relying on a non-zero exit code doesn't #                 distinguish an ephemeral failed call from a "not-exists". if gcloud compute instance-templates describe "$1"; then echo "Instance template ${1} already exists; continuing." >&2 return fi </ins> local attempt=0 while true; do if ! gcloud compute instance-templates create "$1" 

--can-ip-forward  --metadata-from-file "$3" "$4"; then if (( attempt > 5 )); then <del> echo -e "${color_red}Failed to create instance template $1 ${color_norm}" </del> <ins> echo -e "${color_red}Failed to create instance template $1 ${color_norm}" >&2 </ins> exit 2 fi <del> echo -e "${color_yellow}Attempt $(($attempt+1)) failed to create instance template $1. Retrying.${color_norm}" </del> <ins> echo -e "${color_yellow}Attempt $(($attempt+1)) failed to create instance template $1. Retrying.${color_norm}" >&2 </ins> attempt=$(($attempt+1)) else break

echo "Creating minions." <ins> # TODO(mbforbes): Refactor setting scope flags. </ins> local -a scope_flags=() if (( "${#MINION_SCOPES[@]}" > 0 )); then scope_flags=("--scopes" "${MINION_SCOPES[@]}")