Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod got stuck in Pending state forever when failed to bind to host. What you expected to happen: For it to be rescheduled on another host How to reproduce it (as minimally and precisely as possible): Not always reproducible. But simply created an RC replicas=2 on a 3 host setup, and one of the Pods got stuck in pending state for a while. Following error messages were found in the scheduler logs: Describe pod output: RC yml: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): RHEL7, Docker native 1.12.6 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scheduling
Can you provide the details steps for this issue? And the output of ?
The log just showed the node was notReady or unreachable. Seems the kubelet node is not quite stable that the connections with the master always get lost.
Another pod of the same RC (replicas=2) was started successfully on one of the nodes. I'd expect the failed one to at least be attempted to start again, may be on the node where its peer is running (as there are no anti affinity rules defined on it). But it was stuck in pending state forever (and there was no other reference for the failed pod besides the original log I've attached). I was able to create new RCs of the same kind on the same set of nodes successfully after the Pod scheduling failure, so it looks like some nodes were available for allocation. Perhaps there are some other logs I can look at /provide? no definite steps. We just extensive validation tests for kubernetes were a bunch of RCs/Deployments/Services/Ingress controllers get created. And starting k8s 1.7.x started observing this failure. Once we see it again, I'll fetch the node status and update the bug.
From the log, it seemed that failed to bind your pod to the node and also failed to forget the pod, which led to unable to reschedule the pod to other nodes. Would you please append when you start and reproduce this issue. Also the kubelet log of that erroneous node.
^^ could you start the schedule with the above param next time for validation test run?
/cc
I ran our tests again, and hit this issue with several pods: There were a few more. Since none of them is assigned to a node. I included logs for all kubelet's. I also included scheduler log with : :
Seeing this as well. hypothesis: broke the error handling when fails. Before that commit, when failed it would log an error and pass the pod to the error handler (), which is in charge of retrying scheduling the pod. After that commit, when fails it skips the error handling and scheduling is never retried. I'm doing some experiments with patching that and I think there's more to this bug than just that (for example why is failing in the first place? is the scheduler cache corrupted? why?), but that's what I've got so far.
/cc
Can you post the description of per namespace?() Are you enabled ? And what's the config of component?
there are no specific parameters passed on the scheduler start besides address and cloudconfig
I posted a patch that i believe fixes this issue at -- for us it's resolved the issue so far, but I'm not very reliably able to reproduce so it's a bit hard to check. could you apply and see if your validation tests pass?
thanks!! We will be able to validate it either today or early next week
Are you able to validate the fix provided by through ? Just saw this issue, and I think it is a serious regression in 1.7 release and we should patch it. cc/ the patch manager for 1.7 patch release.
our QA have been busy with the current release, and didn't have a chance to validate the fix yet. Hopefully this week. Patching sounds like a great idea. It is a regression indeed as we haven't hit this issue in the previous versions of k8s with the same set of validation tests.
Thanks everyone who helped investigate this!
just ran validation test against the patched branch - no pods stuck in Pending any more, so the fix did the job! thank you
I have this problem with k8s v1.7.0 . How can I fix it, upgrading to v1.7.X? Thank you!
I am seeing this in 1.9. What steps I need to take to solve this problem?
I am seeing this periodically on Azure VMs
any chance you could open a new issue linked to this one with the requisite troubleshooting info (versions, etc.) and label it with ` Thanks!

Discussion:
} // assume signals to the cache that a pod is already in the cache, so that binding can be asnychronous. <del> func (sched *Scheduler) assume(pod *v1.Pod, host string) error { </del> <ins> // assume modifies `assumed`. func (sched *Scheduler) assume(assumed *v1.Pod, host string) error { </ins> // Optimistically assume that the binding will succeed and send it to apiserver // in the background. // If the binding fails, scheduler will release resources allocated to assumed pod // immediately. <del> assumed := *pod </del> assumed.Spec.NodeName = host <del> if err := sched.config.SchedulerCache.AssumePod(&assumed); err != nil { </del> <ins> if err := sched.config.SchedulerCache.AssumePod(assumed); err != nil { </ins> glog.Errorf("scheduler cache AssumePod failed: %v", err) // TODO: This means that a given pod is already in cache (which means it // is either assumed or already added). This is most probably result of a

// predicates in equivalence cache. // If the binding fails, these invalidated item will not break anything. if sched.config.Ecache != nil { <del> sched.config.Ecache.InvalidateCachedPredicateItemForPodAdd(pod, host) </del> <ins> sched.config.Ecache.InvalidateCachedPredicateItemForPodAdd(assumed, host) </ins> } return nil }

// Tell the cache to assume that a pod now is running on a given node, even though it hasn't been bound yet. // This allows us to keep scheduling without waiting on binding to occur. <del> err = sched.assume(pod, suggestedHost) </del> <ins> assumedPod := *pod // assume modifies `assumedPod` by setting NodeName=suggestedHost err = sched.assume(&assumedPod, suggestedHost) </ins> if err != nil { return } // bind the pod to its host asynchronously (we can do this b/c of the assumption step above). go func() { <del> err := sched.bind(pod, &v1.Binding{ ObjectMeta: metav1.ObjectMeta{Namespace: pod.Namespace, Name: pod.Name, UID: pod.UID}, </del> <ins> err := sched.bind(&assumedPod, &v1.Binding{ ObjectMeta: metav1.ObjectMeta{Namespace: assumedPod.Namespace, Name: assumedPod.Name, UID: assumedPod.UID}, </ins> Target: v1.ObjectReference{ Kind: "Node", Name: suggestedHost,

sendPod          *v1.Pod algo             algorithm.ScheduleAlgorithm expectErrorPod   *v1.Pod <ins> expectForgetPod  *v1.Pod </ins> expectAssumedPod *v1.Pod expectError      error expectBind       *v1.Binding

expectAssumedPod: podWithID("foo", testNode.Name), injectBindError:  errB, expectError:      errB, <del> expectErrorPod:   podWithID("foo", ""), </del> <ins> expectErrorPod:   podWithID("foo", testNode.Name), expectForgetPod:  podWithID("foo", testNode.Name), </ins> eventReason:      "FailedScheduling", }, { sendPod:     deletingPod("foo"),

for i, item := range table { var gotError error var gotPod *v1.Pod <ins> var gotForgetPod *v1.Pod </ins> var gotAssumedPod *v1.Pod var gotBinding *v1.Binding configurator := &FakeConfigurator{ Config: &Config{ SchedulerCache: &schedulertesting.FakeCache{ <ins> ForgetFunc: func(pod *v1.Pod) { gotForgetPod = pod }, </ins> AssumeFunc: func(pod *v1.Pod) { gotAssumedPod = pod },

if e, a := item.expectErrorPod, gotPod; !reflect.DeepEqual(e, a) { t.Errorf("%v: error pod: wanted %v, got %v", i, e, a) } <ins> if e, a := item.expectForgetPod, gotForgetPod; !reflect.DeepEqual(e, a) { t.Errorf("%v: forget pod: wanted %v, got %v", i, e, a) } </ins> if e, a := item.expectError, gotError; !reflect.DeepEqual(e, a) { t.Errorf("%v: error: wanted %v, got %v", i, e, a) }

// FakeCache is used for testing type FakeCache struct { AssumeFunc func(*v1.Pod) <ins> ForgetFunc func(*v1.Pod) </ins> } func (f *FakeCache) AssumePod(pod *v1.Pod) error {

func (f *FakeCache) FinishBinding(pod *v1.Pod) error { return nil } <del> func (f *FakeCache) ForgetPod(pod *v1.Pod) error { return nil } </del> <ins> func (f *FakeCache) ForgetPod(pod *v1.Pod) error { f.ForgetFunc(pod) return nil } </ins> func (f *FakeCache) AddPod(pod *v1.Pod) error { return nil }