Change:
This test is currently failing in sig-release master-upgrade suite, This could potentially become release blocking if not addressed sooner. can you please triage this failure? /kind bug /priority failing-test /priority important-soon /sig testing cc /assign
hmm, looks like cluster up etc works but some of the test case(s) are hanging and taking way too much time: ... I0417 18:59:21.467] (Will retry.)
cc ^^ assume that's somewhere inside the upgrade framework
/sig cluster-lifecycle
/assign /unassign Unlikely I will have the time or expertise to patch these, discussed out of band with and these appear to be something in the upgrade or cluster-up code, not kubetest etc..
/milestone v1.11
Debugged one of the jobs. The node upgrade is stalling out. There's logic in the node upgrade to wait for the new GCE vm to register with the k8s API. It does this by checking the node status's external ID and seeing if it matches the instance ID from the GCE vm. That logic is here: It looks like the kubelet is now registering the GCE VM's resource path and not its ID as the node's externalID. from https://k8s- Can I get input on this from ? Is this an intended change? If so, what is a good way to detect that the new node has registered with kubernetes?
cc ^^ :-)
ExternalID was apparently deprecated before kubernetes V1 API, but was never removed properly. PR was merged recently to remove the use of ExternalID Can we use the instance ID instead?
No. Instance ID is returning information like "k8s-jkns-e2e-gce-slow/us-central1-f/bootstrap-e2e-minion-group-0s1q". During a node upgrade, this ID is shared between the old and new VM. We need a way to distinguish them to make sure the new VM has come up, registered with the API server, and is healthy. Without a distinguishing ID, we could be reading the health status of the old node before we deleted it.
Ah, didn't read the issue carefully. Thanks for explaining that. Kubelet also reports machine ID/ system UUID. Are they unique in the cloud environment?
Can you point to where I can find that in the API and save me having to track it down? On Tue, Apr 17, 2018, 17:24 Yu-Ju Hong <:
or
Sorry, I should have been more specific. Where do those come from? Because I can't find that anywhere in
I think those are from . I don't think they are exposed in the gcloud. If we do use them, I think the upgrade script will have to simply check if the machine ID has changed, hmm...
Oh I see. OK I'll see if I can work with that.
It doesn't work. The machine ID is the same before and after the instance recreate. Even confirmed by looking at directly. seems to have the same value as machine ID.
Urghh...recreating an instance actually keeps the machineID. I don't have a good solution. One idea is to check if the bootID changed.
status/approved-for-milestone
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/reopen I still see the Upgrade test failing in the jobs below.
I still see UpgradeTest and Timeout failing in the jobs below. Are you actively investigating these or should I re-triage?
These failures seem similar to my comment in I'm would appreciate help in looping in the node team on why that is happening.
/cc can either of you please help investigate the node side of this issue to unblock This is currently on of the few tests failing our sig-release upgrade blocking suite. If this is not fixed we will have to delay beta for 1.11 which is slated for 5/15.
/cc
cc
I updated the issue at , which might explain some of the failures here.
I see the tests passing now. Thanks and for all your help.

Discussion:
# Note: This is called multiple times from do-node-upgrade() in parallel, so should be thread-safe. function do-single-node-upgrade() { local -r instance="$1" <del> instance_id=$(gcloud compute instances describe "${instance}"  --format='get(id)'  --project="${PROJECT}"  --zone="${ZONE}" 2>&1) && describe_rc=$? || describe_rc=$? if [[ "${describe_rc}" != 0 ]]; then echo "== FAILED to describe ${instance} ==" echo "${instance_id}" return ${describe_rc} </del> <ins> local kubectl_rc local boot_id=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output=jsonpath='{.status.nodeInfo.bootID}' 2>&1) && kubectl_rc=$? || kubectl_rc=$? if [[ "${kubectl_rc}" != 0 ]]; then echo "== FAILED to get bootID ${instance} ==" echo "${boot_id}" return ${kubectl_rc} </ins> fi # Drain node echo "== Draining ${instance}. == " >&2 <ins> local drain_rc </ins> "${KUBE_ROOT}/cluster/kubectl.sh" drain --delete-local-data --force --ignore-daemonsets "${instance}"  && drain_rc=$? || drain_rc=$? if [[ "${drain_rc}" != 0 ]]; then

# Recreate instance echo "== Recreating instance ${instance}. ==" >&2 <del> recreate=$(gcloud compute instance-groups managed recreate-instances "${group}"  </del> <ins> local recreate_rc local recreate=$(gcloud compute instance-groups managed recreate-instances "${group}"  </ins> --project="${PROJECT}"  --zone="${ZONE}"  --instances="${instance}" 2>&1) && recreate_rc=$? || recreate_rc=$?

return ${recreate_rc} fi <del> # Wait for instance to be recreated echo "== Waiting for instance ${instance} to be recreated. ==" >&2 while true; do new_instance_id=$(gcloud compute instances describe "${instance}"  --format='get(id)'  --project="${PROJECT}"  --zone="${ZONE}" 2>&1) && describe_rc=$? || describe_rc=$? if [[ "${describe_rc}" != 0 ]]; then echo "== FAILED to describe ${instance} ==" echo "${new_instance_id}" echo "  (Will retry.)" elif [[ "${new_instance_id}" == "${instance_id}" ]]; then echo -n . else echo "Instance ${instance} recreated." break fi sleep 1 done # Wait for k8s node object to reflect new instance id </del> <ins> # Wait for node status to reflect a new boot ID. This guarantees us # that the node status in the API is from a different boot. This # does not guarantee that the status is from the upgraded node, but # it is a best effort approximation. </ins> echo "== Waiting for new node to be added to k8s.  ==" >&2 while true; do <del> external_id=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output=jsonpath='{.spec.externalID}' 2>&1) && kubectl_rc=$? || kubectl_rc=$? </del> <ins> local new_boot_id=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output=jsonpath='{.status.nodeInfo.bootID}' 2>&1) && kubectl_rc=$? || kubectl_rc=$? </ins> if [[ "${kubectl_rc}" != 0 ]]; then echo "== FAILED to get node ${instance} ==" <del> echo "${external_id}" </del> <ins> echo "${boot_id}" </ins> echo "  (Will retry.)" <del> elif [[ "${external_id}" == "${new_instance_id}" ]]; then </del> <ins> elif [[ "${boot_id}" != "${new_boot_id}" ]]; then </ins> echo "Node ${instance} recreated." break <del> elif [[ "${external_id}" == "${instance_id}" ]]; then echo -n . </del> else <del> echo "Unexpected external_id '${external_id}' matches neither old ('${instance_id}') nor new ('${new_instance_id}')." echo "  (Will retry.)" </del> <ins> echo -n . </ins> fi sleep 1 done

# Ready=True. echo "== Waiting for ${instance} to become ready. ==" >&2 while true; do <del> cordoned=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output='jsonpath={.status.conditions[?(@.type == "SchedulingDisabled")].status}') ready=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output='jsonpath={.status.conditions[?(@.type == "Ready")].status}') </del> <ins> local cordoned=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output='jsonpath={.status.conditions[?(@.type == "SchedulingDisabled")].status}') local ready=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output='jsonpath={.status.conditions[?(@.type == "Ready")].status}') </ins> if [[ "${cordoned}" == 'True' ]]; then echo "Node ${instance} is still not ready: SchedulingDisabled=${ready}" elif [[ "${ready}" != 'True' ]]; then