Code change:
getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.

Related discussion:
text: return false, false, false, nil } <del> // TODO: Move it to the predicates for _, c := range node.Status.Conditions { if critical { break } // TODO: There are other node status that the DaemonSet should ideally respect too, //       e.g. MemoryPressure, and DiskPressure if c.Type == v1.NodeOutOfDisk && c.Status == v1.ConditionTrue { // the kubelet will evict this pod if it needs to. Let kubelet // decide whether to continue running this pod so leave shouldContinueRunning // set to true shouldSchedule = false } } </del> reasons, nodeInfo, err := dsc.simulate(newPod, node, ds) if err != nil { glog.Warningf("DaemonSet Predicates failed on node %s for ds '%s/%s' due to unexpected error: %v", node.Name, ds.ObjectMeta.Namespace, ds.ObjectMeta.Name, err)