Change:
} // Pop removes the head of the active queue and returns it. It blocks if the <del> // activeQ is empty and waits until a new item is added to the queue. It also // clears receivedMoveRequest to mark the beginning of a new scheduling cycle. </del> <ins> // activeQ is empty and waits until a new item is added to the queue. </ins> func (p *PriorityQueue) Pop() (*v1.Pod, error) { p.lock.Lock() defer p.lock.Unlock()

Discussion:
What happened: Kubernetes scheduler may leave a single pod stuck in pending state after a worker node is reloaded/reinstalled. This problem appears to have been introduced in Kubernetes v1.12.4 and v1.13.1. We have not been able to recreate the problem in our testing on earlier releases. The key to the failure is that the worker being reloaded/reinstalled must be the only scheduling option available. The pending pod is eventually scheduled if a new scheduler leader is elected or the node status changes. This appears to be a timing bug since the problem does not always occur. What you expected to happen: No pods are stuck in pending state. How to reproduce it (as minimally and precisely as possible): Start with a single node cluster running a handful of pods. Cordon, drain and then reload/reinstall the node. Since this is a timing bug, it may take several iterations to recreate the problem. Environment: Kubernetes version (use ): 1.12.4, 1.13.1 and 1.13.2 Cloud provider or hardware configuration: VMs running in VirtualBox OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Custom ansible scripts Others: N/A
/sig scheduling
Events from the pending pod: Events from the node:
does "reload/reinstall" mean info is still kept in etcd? or (or equivalent steps) is executed to remove that node entirely? And I assume the "reloaded" node keeps the same spec and IP/node name with previous one?
the info is still kept in etcd. And the node keeps the same spec, IP and name as it had before. The reload does the following: and drain the node the VM the VM again with the same spec the node
What's the taint/toleration accordingly?
there were no taints to the node during this process. The pods are part of a deployment with only basic tolerations provided by Kubernetes. Deleting the pod will get it scheduled even though the taints and tolerations are unchanged. Simply restarting the scheduler leader will get the pod scheduled as well.
Can you share and ?
The symptom (esp. "a single pod stuck in pending state") is quite similar as issue .

I think why this occurs since v1.12.4 is related the PR - which was delivered to master branch, and backported to v1.12.4, and it appears to reveal higher chance of racing. cc/ Basically we have 2 solutions to fix this: the real root cause of the racing condition and give fix (I haven't got luck to reproduce this yet) that we have a solution () to alleviate this kind of problem, could you try to run your test on a) master branch or b) v1.12.4 with cherrypicked to see if the issue is resolved? and does this fits your business need? (in other words, the single pod may be pending for 30 seconds at most)
I'll recreate and provide you the details. However, the pod that is stuck pending is different every time.
Thanks, I'll work on the recreate and test of the cherrypick. That PR certainly seems promising.
Node YAML after recreate:
Pod YAML after recreate:
the cherrypick doesn't look trivial for 1.12 or 1.13. Give me some time to setup an environment to test on 1.14. Unless you have another way?
Yeap, it's not a trivial task to rebuild the source code. Let me know if you need any help. I suppose you're going to try on latest 1.14 tag? e.g. v1.14.0-alpha.1? Right now I'm working on the 1st solution I mentioned in The challenge is I can't reproduce this in my local dev env (I'm running on source code branch of v1.12.4) yet, so my current plan is to retry reproducing your issue to make my solution more convincing. If still no luck, I will send out the PR for reviewing early next week.
And let me confirm with your statement: When the issue occurs, everytime there is only one single pod stuck in pending state, right?
Still don't have luck in reproducing the issue. But I've prepared a PR and put detailed analysis there. Any comments are welcome.
No, the problem does not occur every time. It can take as many as 7 retries before reproducing it.

Sry, I should have made it more clear - I wanted to confirm that among all the times the issue is reproducible, is that everytime there is only one single pod in pending state?
correct, every time that I've reproduced the problem there has been only one pod in pending state.
Thanks. To some extent, it's consistent with the deduction in my PR.
is it possible to have the necessary fixes cherry-picked to at least 1.12 and 1.13? That would make it easier for me to build and test?
Yeap, cherrypick PRs ( onto 1.13, onto 1.12) were created and will be reviewed/approved soon.
Thank you for the cherrypick PRs. I rebuilt 1.13 with the cherrypick PR and still saw the problem on 2 of 10 recreate attempts. Without the fix, I saw a similar failure rate.
the fix is only ready in master right now. Did you rebuild 1.13 by manually running “git cherrypick <this commit id” on release-1.13? Or directly built from latest src code of release-1.13?
I manually rebuilt 1.13.2 with
/reopen I will take another look.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I've verified is able to fix your issue thoroughly. And cherrypick PRs have been raised (1.12 , 1.13 ).
Thank you.
Thanks, and K8s 13.3 already has another change () that should alleviate the problem significantly. K8s 13.4 will have the definitive fix. Given that the root-cause is fixed, we can close this issue.