Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: kube-scheduler crashes (panics) in rare circumstances. Stack trace: What you expected to happen: flawless operation ;) How to reproduce it (as minimally and precisely as possible): Unclear. The error above looks like it was associated with some cloning of node metadata. Perhaps due to a race during node scale-up/scale-down and cache interactions. Anything else we need to know?: Environment: Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig scheduling
are you using hostPort for your workload?
is there a specific workload you had in mind? The does, as well as daemonsets for our service mesh, which are in host networking mode.
There was a PR () in v1.12/v1.11.3 fixing issue when workloads uses hostNetwork, and the solution is to correct the NodeInfo cloning logic. So it's quite likely related with the symptom you described. Could you give a try on v1.11.3?
BTW: the issue that PR fixed is only reproducible on some circumstances.
Thanks - I'll give this a whirl. I think I can optimistically close this issue - we'll reopen if it persists with a more recent autoscaler release. Cheers!
Sounds good! Thanks for the feedback.
we just had a similar issue this morning and our cluster is running kube 1.11.4: and these are the error logs : we do have hostNetwork work load running inside our cluster but not sure if that's caused by other issues since you mentioned that hostNetwork issue has been fixed since 1.11.3.
could you re-open this issue ? or I can raise a new issue for it.
/reopen I will take a look.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks
are the worker nodes auto scaled out/in? e.g. by an auto-scaler component?
yes, we do have an autoscaler to auto scale our worker nodes and we are using since most of our workloads are job based.
I see. It's likely an internal cache inconsistency. (not related with the hostPort issue I fixed earlier) /assign
One thing I'm curious is from the log it points to L386 of Then it means is nil, but should't give a panic trace at the first call inside ? Say: I looked through the code, but can't find a place that could be nil. So are you using raw v1.11.4-beta.0 branch, or did some modification? As in my opinion, it makes sense the panic comes from L385:
No, in Go if n.requestedResource is nil, you will get a stack trace similar to what is given in this issue.
I read it differently. I wrote an example , it shows deepest trace at L12, rather than L17.
I was actually wondering if it's possible that the stack trace isn't 100% line accurate.. Either that it systematically hits the next line in this case, or that the order in which field copies are executed isn't a simple 1:1 (linear) relationship to the code.
it's interesting, maybe it's some trick (or bug) in apimachinery's runtime.HandleCrash()? And I wrote a test to reproduce the issue, even the same line on the deepest stack trace: (although I think it should be line above)
/reopen I will raise cherrypick PRs to 1.12 and 1.11, and close this issue by then.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi i have scaled my cluster to 100 node from 40 nodes as there were 400 pending pods and for some of the nodes all pods get schedule properly while around 28 nodes have only dnsmasq and kube-proxy was running. after around 40 min those node had weave and other pods started running. i coundn't able to find any other issue in my cluster apart from kube-scheduler got restarted because same error mentino in the issue. though it restarted in few second. i was just wondering that if after restart would it takes 30-40 min to schedule pods? cause after 40 min everything seems to be working properly. I am running kubernetes 1.12.1 and i have installed it using kops 1.10.0
fix of this issue is cherrypicked (PR ) into v1.12.4. Could you give a try on v1.12.4?
Close this as fix has been applied to 1.11+ k8s versions. /close
Closing this issue. <detailsIn response to : Close this as fix has been applied to 1.11+ k8s versions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
queue internalqueue.SchedulingQueue, pdbs []*policy.PodDisruptionBudget, ) (map[*v1.Node]*schedulerapi.Victims, error) { nodeToVictims := map[*v1.Node]*schedulerapi.Victims{} var resultLock sync.Mutex

queue internalqueue.SchedulingQueue, pdbs []*policy.PodDisruptionBudget, ) ([]*v1.Pod, int, bool) { <ins> if nodeInfo == nil { return nil, 0, false } </ins> potentialVictims := util.SortableList{CompFunc: util.HigherPriorityPod} nodeInfoCopy := nodeInfo.Clone()

test.predicates[algorithmpredicates.MatchInterPodAffinityPred] = algorithmpredicates.NewPodAffinityPredicate(FakeNodeInfo(*nodes[0]), schedulertesting.FakePodLister(test.pods)) } nodeNameToInfo := schedulercache.CreateNodeNameToInfoMap(test.pods, nodes) <ins> // newnode simulate a case that a new node is added to the cluster, but nodeNameToInfo // doesn't have it yet. newnode := makeNode("newnode", 1000*5, priorityutil.DefaultMemoryRequest*5) newnode.ObjectMeta.Labels = map[string]string{"hostname": "newnode"} nodes = append(nodes, newnode) </ins> nodeToPods, err := selectNodesForPreemption(test.pod, nodeNameToInfo, nodes, test.predicates, PredicateMetadata, nil, nil) if err != nil { t.Error(err)