Change:
https://k8s- /kind bug /priority failing-test /priority important-soon /sig testing /milestone v1.11 cc /assign for triage
GitHub didn't allow me to assign the following users: for, triage. Note that only and repo collaborators can be assigned. <detailsIn response to : ## Failing Job https://k8s- /priority failing-test /priority important-soon /sig testing /milestone v1.11 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is sig-testing. Looks like , , /unassign /remove-sig testing
/sig cluster-lifecycle /assign can you see if this is a cluster lifecycle issue please?
Chiming in for cluster-lifecycle, this is weird. Somehow the nodes are being put back into a cordoned state after the upgrade. The upgrade checks that after a node is upgraded, it becomes both ready and schedulable. Here's an example from the linked failure: But just a few lines below that, we can see the the cluster validation after the upgrade shows that 2 nodes are still unschedulable. Does someone from the node team know why this might be happening? Is it a race condition? It worked for one node, but not two in this case. In other runs, I see it working for two nodes and only leaving one unschedulable. cc
/cc can either of you please help investigate the node side of this issue to unblock This is currently on of the few tests failing our sig-release upgrade blocking suite. If this is not fixed we will have to delay beta for 1.11 which is slated for 5/15.
/cc
cc
/cc I am also running into this issue when trying to add a new upgrade test. First, I think the reason that the test log said the node was ready is because the code is looking at a wrong field for SchedulingDisabled. The unschedulable field is a field in not a node status condition, so I think we should instead use: local cordoned={KUBE_ROOT}" get node "${instance}" --output='jsonpath=="{.spec.unschedulable}"' I am also puzzled on why recreating instance just rebooted the node. According to 'gcloud instance-groups managed recreate-instances' doc, that command should delete the instance first and create a new one. For the passed cluster upgrade test, that seems the case, but for the failed ones, I saw node was just rebooted and so still got the old unschedulable state from the API server. While trying to figure out why recreating instance just rebooted node sometimes, would it make sense we explicitly use 'gcloud instance-groups managed delete-instances' and 'gcloud instance-groups managed create-instances' ?
We use recreate-instances to maintain the node identity and name. How are you determining that it just rebooted and not recreated? Did the nodes stop recreating their node objects on reboot and no longer clear out the field? Is this a consequence of ? Since is no longer unique across , the node object doesn't get recreated? Would that mean the upgrade script has to uncordon the node now?
I think theory at is right. After was merged, .spec.externalID is mapping to spec.instanceID which wouldn't be unique after recreate. In this case, Kubelet won't delete old node object and recreate new one. I think this is what we want: we want to stop the node to delete and create itself, we want to stop the node to mess up its taints and labels. Instead, we want the controller(s) to explicitly set the taints and labels to a node. also updated upgrade script to use boot id to detect the new boot or recreation: In this very case, we should update upgrade script to explicitly clear the unschedulable field once new boot id was detected.
Thanks, I'll work on a PR and send it out today.
From my manual run, I ssh-ed into the unschedulable node after the test and saw the kubelet log contains multiple restarts and logged the event that the node was previously registered and then later changed to NodeNotSchedulable based on the spec got from the API server: 1754: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Node e2e-test-jiayingz-minion-group-ssbv was previously registered 1755: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Successfully registered node e2e-test-jiayingz-minion-group-ssbv ... 1789: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"e2e-test-jiayingz-minion-group-ssbv", UID:"e2e-test-jiayingz-minion-group-ssbv", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeNotSchedulable' Node e2e-test-jiayingz-minion-group-ssbv status is now: NodeNotSchedulable
That fits with the current theory. The instance is being recreated, but the kubelet is no longer re-registering itself since the externalID is the same.
/cc
was sufficient to fix this issue or are we also waiting on to b merged? I am still waiting for the latest run to verify if the issue is fixed.
/reopen There's a bug with my fix. It fails if the uncordon command happens in the window where the node object doesn't exist. I will have a fix out shortly.
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen There's a bug with my fix. It fails if the uncordon command happens in the window where the node object doesn't exist. I will have a fix out shortly. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
Note that this bug is flaky. It doesn't happen 100% of the time. Looks like we got 100% pass in the latest run: https://k8s- Here's an upgrade test that's hitting this flake: https://k8s-
cc Please see my comments at for the change Before the change, when instance is recreated, kubelet will delete the node object and a new one will be created. Now, kubelet will not delete the old one if the node name is the same, I think. This might give some trouble for volumes because the old node will contain the outdated volumesAreAttached information. But we do sync up with the cloud provider and update this information eventually. I am not sure whether this old object might cause other problems or not.
Can you move that to another issue? It is a legitimate bug that the upgrade logic should not rely on kubelet recreating the node object. I agree that no longer doing that may lead to other issues, but that should be a separate discussion.
I see the test failing in the last few runs. Is it a different issue plaguing the test now? https://k8s-
/reopen
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 1 day, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Here are the since the last failure.
Spoke to offline and SkewTest is a overall for the second half of the suite. So if the other tests in the job pass, this one should turn the corner. so closing this bug for now.

Discussion:
sleep 1 done <del> # Wait for the node to not have SchedulingDisabled=True and also to have # Ready=True. </del> <ins> # Uncordon the node. echo "== Uncordon ${instance}. == " >&2 local uncordon_rc "${KUBE_ROOT}/cluster/kubectl.sh" uncordon "${instance}"  && uncordon_rc=$? || uncordon_rc=$? if [[ "${uncordon_rc}" != 0 ]]; then echo "== FAILED to uncordon ${instance} ==" return ${uncordon_rc} fi # Wait for the node to have Ready=True. </ins> echo "== Waiting for ${instance} to become ready. ==" >&2 while true; do <del> local cordoned=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output='jsonpath={.status.conditions[?(@.type == "SchedulingDisabled")].status}') </del> local ready=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output='jsonpath={.status.conditions[?(@.type == "Ready")].status}') <del> if [[ "${cordoned}" == 'True' ]]; then echo "Node ${instance} is still not ready: SchedulingDisabled=${ready}" elif [[ "${ready}" != 'True' ]]; then </del> <ins> if [[ "${ready}" != 'True' ]]; then </ins> echo "Node ${instance} is still not ready: Ready=${ready}" else echo "Node ${instance} Ready=${ready}"