Change:
Is this a BUG REPORT or FEATURE REQUEST?: What happened: admission control plugin in apiserver. a deployment with 1 replica. The pod is successfully launched. By default the admission control plugin will add following toleration in the pod. the node where the above pod is running. of image locality policy and above toleration in the pod, scheduler still choose the node to launch the pod. The pod is in state and failed to create. What you expected to happen: The pod should be scheduled to healthy node and created successfully. How to reproduce it (as minimally and precisely as possible): See above steps. Anything else we need to know?: Environment: Kubernetes version (use ): v1.10 Cloud provider or hardware configuration: openstack OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-121-generic Install tools: N/A Others:
/sig scheduler
/sig scheduling /sig node
I think that we should avoid scheduling pods to nodes which is not ready even it can tolerate and , comments
As the pod already claimed to tolerate NotReady, this is an expected behavior. P.S. If the eviction dose not happen after 5m, that should be the bug. Please reopen the issue and modify the description. This is what tolerations is designed for :) And there are many use cases for pod to be scheduled to NotReady nodes (mostly during cluster deployment).
I think maybe you can configure effect to avoid scheduling to those nodes.
scheduler still choose the NotReady node to launch the pod For running, the toleration is used for graceful termination by ; if is enabled, it's better to also taint node with for .
Wondering how the pod can start to run if the Node is not ready at deployment time? Can you specify a use case here?
It won't need to start, system pod pending will be expected :) The point is, freedom for users to decide what tolerations they want matters here. I don't think it's reasonable to combine with in this case.
Perhaps, add to default toleration of admission control plugin could eliminate the confusion.
We should align the behavior when enabled or disabled : if is disabled, will not schedule pod to not-ready node. if is enabled, we should also taint node with . with timeout is used for graceful termination of 'running pods' for feature. We should not re-schedule pod to not-ready unless tolerating it explicitly.
reopen this for more discussion :)
Let me extracted the mixed problems in this issue :-) First one: These two default tolerations attached to the pod by means: if the node is tainted as or , wait for 300s before the pod is evicted from the node. While according to the issue description, is working as expected (pods are not evicted immediately). This part should be closed. Second one: The discussion raised is actually about taint during eviction: if TaintNodesByCondition is enabled, we should also taint node with NotReady:NoSchedule. My thoughts is: it's reasonable to taint node with as well for consistency consideration (but I'm hesitate if only for eliminating confusion as pods are expected to pending on NotReady nodes) I think it should be clearer now.
I open a PR If TaintNodesByCondition is enable, taint node with NotReady:NoSchedule and Unreachable:NoSchedule.
, we did not catch 1.11 release, so this issue will be handled in 1.12 :)
in 1.9.7enable the TaintNodesByCondition and taint nodee.g. node01 with NotReady:NoSchedulethen pod also can not be scheduled to node01 even when node01 is healthy. What you expected to happen: The pod should be scheduled to node01 when it is healthy node and not be scheduled to node01 when it is not ready. Thanks
How did you do that? by kubectl? The fix is not ready until 1.12.
YesI tried "kubectl taint nodes node01 ",and also tried "kubectl taint nodes node01 "
this feature did not handle taint manually
Ok, thanks a lot! So I can not solve the problem manually now and what I should do is just waiting for the version 1.12, right?

Discussion:
Effect: v1.TaintEffectNoExecute, } <del> nodeConditionToTaintKeyMap = map[v1.NodeConditionType]string{ v1.NodeMemoryPressure:     algorithm.TaintNodeMemoryPressure, v1.NodeOutOfDisk:          algorithm.TaintNodeOutOfDisk, v1.NodeDiskPressure:       algorithm.TaintNodeDiskPressure, v1.NodeNetworkUnavailable: algorithm.TaintNodeNetworkUnavailable, v1.NodePIDPressure:        algorithm.TaintNodePIDPressure, </del> <ins> nodeConditionToTaintKeyStatusMap = map[v1.NodeConditionType]struct { taintKey string // noScheduleStatus is the condition under which the node should be tainted as not schedulable for this // NodeConditionType noScheduleStatus v1.ConditionStatus }{ v1.NodeReady: { taintKey:         algorithm.TaintNodeNotReady, noScheduleStatus: v1.ConditionFalse, }, v1.NodeMemoryPressure: { taintKey:         algorithm.TaintNodeMemoryPressure, noScheduleStatus: v1.ConditionTrue, }, v1.NodeOutOfDisk: { taintKey:         algorithm.TaintNodeOutOfDisk, noScheduleStatus: v1.ConditionTrue, }, v1.NodeDiskPressure: { taintKey:         algorithm.TaintNodeDiskPressure, noScheduleStatus: v1.ConditionTrue, }, v1.NodeNetworkUnavailable: { taintKey:         algorithm.TaintNodeNetworkUnavailable, noScheduleStatus: v1.ConditionTrue, }, v1.NodePIDPressure: { taintKey:         algorithm.TaintNodePIDPressure, noScheduleStatus: v1.ConditionTrue, }, </ins> } taintKeyToNodeConditionMap = map[string]v1.NodeConditionType{ <ins> algorithm.TaintNodeNotReady:           v1.NodeReady, </ins> algorithm.TaintNodeNetworkUnavailable: v1.NodeNetworkUnavailable, algorithm.TaintNodeMemoryPressure:     v1.NodeMemoryPressure, algorithm.TaintNodeOutOfDisk:          v1.NodeOutOfDisk,

func (nc *Controller) doNoScheduleTaintingPass(node *v1.Node) error { // Map node's condition to Taints. <del> taints := []v1.Taint{} </del> <ins> var taints []v1.Taint </ins> for _, condition := range node.Status.Conditions { <del> if _, found := nodeConditionToTaintKeyMap[condition.Type]; found { if condition.Status == v1.ConditionTrue { </del> <ins> if taint, found := nodeConditionToTaintKeyStatusMap[condition.Type]; found { if condition.Status == taint.noScheduleStatus { </ins> taints = append(taints, v1.Taint{ <del> Key:    nodeConditionToTaintKeyMap[condition.Type], </del> <ins> Key:    taint.taintKey, </ins> Effect: v1.TaintEffectNoSchedule, }) }

Key:    algorithm.TaintNodeNetworkUnavailable, Effect: v1.TaintEffectNoSchedule, } <ins> notReadyTaint := &v1.Taint{ Key:    algorithm.TaintNodeNotReady, Effect: v1.TaintEffectNoSchedule, } </ins> tests := []struct { Name           string

}, ExpectedTaints: []*v1.Taint{networkUnavailableTaint}, }, <ins> { Name: "Ready is false", Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionFalse, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, ExpectedTaints: []*v1.Taint{notReadyTaint}, }, </ins> } for _, test := range tests {

} } <del> // Case 4: Schedule Pod with NetworkUnavailable toleration. </del> <ins> // Case 4: Schedule Pod with NetworkUnavailable and NotReady toleration. </ins> networkDaemonPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "network-daemon-pod",

Operator: v1.TolerationOpExists, Effect:   v1.TaintEffectNoSchedule, }, <ins> { Key:      algorithm.TaintNodeNotReady, Operator: v1.TolerationOpExists, Effect:   v1.TaintEffectNoSchedule, }, </ins> }, }, }