Change:
This is to track the progress of work on moving pod priority and preemption to Beta. /kind feature [x] Write a design doc on how to support preemption in DaemonSet controller. [x] Remove DaemonSet controller scheduling logic and let DaemonSet Pods be schedule by default scheduler. [ ] Add support for multi-scheduler preemption to default scheduler (this is targeted for 1.11). [x] Ensure that cluster setup tools start scheduler as a part of cluster bring-up. This is needed for scheduling DaemonSet Pods. [x] Change nominated-node-name annotation to a field in PodStatus. [x] Add metrics for number of preemptions and number of victims. [x] Change critical pods’ template to use priority [x] Get rid of the rescheduler [x] Write a doc on how Kubelet evictions work with priority [x] Update the comment of PriorityClassName and also the priority design doc to describe "system" priorities. [x] Enable priority and preemption in our existing scalability tests. [x] Move apis/scheduling/v1alpha1 to apis/scheduling/v1beta1 [x] Write a roll out plan /sig scheduling /assign
/cc
and Do you guys think you have time to work on any of the items here? I have written a design doc on supporting preemption in DS controller which I am going to share with all soon and then we can decide which solution to pick and how to proceed there. Adding metrics for preemption is something that would be great if you could help.
Do we already have a doc on how Kubelet eviction works with priority? If not, do you think you can add one?
I it to the eviction design docs: We may want to add it to the open-source documentation as well:
Thanks! Yes, it would be great if you could update the user docs as well.
sure and I can surely work on some of the items.
I will go through your design doc soon,
- I will also go through the docs. I believe I can get started on adding metrics preemptions in meanwhile.
/cc
Can you explain "Get rid of the rescheduler" means?
this rescheduler: which is based on critical pod annotation, as the same functionality will be handled by priority and preemption feature.
I am off next week but I will get started on following items Change critical pods’ template to use priority Get rid of the rescheduler unless someone has already started on it.
For critical pod's template to use priority & , I am planning to do following things: Update IsCriticalPod f(n) to use priorityClass and other relevant changes. Add e2e test. In priority admission controller do a automatic assignment of , in case the pod has just critical pod annotation set(for backwards compatibility). Add deprecation note to section and pod tolerations in section of and add is needed for critical pods. Please let me know, if I have missed any item.
Thanks for the update and your help. Your plan sounds good to me. I have a couple of small comments: think should return true for both and priorities. We should create a function to check these two priorities and returns true for system critical pods. The function should be usable by other components. So, maybe apis/core/v1/helpers is the right place. also think that we should probably assign instead of automatically for critical pods. is a bit lower priority than , so it may cause fewer surprises, yet it is high enough to not get preempted under normal circumstances.
The design doc how to handle preemption for DaemonSet Pods is available . The document includes tasks and their release target.
I have some free time in the fowlling weeks, fell free let me know what can I help here.
FYI, I have identified the following system critical add-ons so far that need to have a critical priority: Kube-dns: system-cluster-critical Kube-proxy: system-node-critical Kubernetes-dashboard: system-cluster-critical fluentd: system-node-critical ip-masq-agent: system-node-critical nvidia-gpu-device-plugin: system-node-critical metadata-proxy: system-node-critical metrics-server: system-cluster-critical heapster: system-cluster-critical influxGrafana: system-cluster-critical calico-node: system-node-critical kube-dns-autoscaler: system-cluster-critical
Rollout plan for Priority and Preemption can be . As usual, you need be a member of to gain access to the doc.
In the self-hosting kubernetes, should the tool like kube-adm need to add system-node-priority to etcd, apiserver pods etc, so that scheduler won't preempt them?
etcd, apiserver, etc usually run on the master node and the master node is marked unschedulable. So, the scheduler shouldn't preempt any pods on the master node. Besides, those pods do not go through admission controllers AFAIK. So, even if we add PriorityClass to those pods, their integer value of priority won't be populated and their priority won't be visible to scheduler.
PSA CNI-providers: this will eventually be a default for critical pods, which would affect your deployments.
adding /priority important-soon to keep tracking this for 1.10 /priority important-soon Please break out the items from this tracker that will actually be to 1.10, and add each of those issues/PRs to the 1.10 milestone with other appropriate labels. Thanks!
If you're going to continue to use this tracker as a way of tracking what's going in 1.10, please add status/approved-for-milestone label ASAP. Thanks!
I changed the milestone. We won't be able to test all pieces of this work with real workloads in time for 1.10.
ref/ kubernetes/website
I think and are remaining PRs for moving to beta. Are the following items needed now: Add support for multi-scheduler preemption to default scheduler (this is targeted for 1.11). Ensure that cluster setup tools start scheduler as a part of cluster bring-up. This is needed for scheduling DaemonSet Pods. Enable priority and preemption in our existing scalability tests. If yes, do we have PRs for them or they need to be created?
This is not critical. We can move it to Beta without this, but we should add it soon. This is critical, but looks like cluster bring up tools already do this. Another round of review is needed to ensure that we are fine here. This should be done. We don't have a PR for this.
I did a grep to find list of files containing rescheduler. Identified the following: CHANGELOG- CHANGELOG- CHANGELOG- cluster/gce/gci/configure- cluster/gce/config- cluster/gce/config- cluster/gce/manifests/kube-proxy.manifest cluster/gce/manifests/BUILD cluster/gce/manifests/rescheduler.manifest cluster/addons/fluentd-elasticsearch/fluentd-es- cluster/addons/fluentd-gcp/fluentd-gcp-configmap- cluster/addons/fluentd-gcp/fluentd-gcp- cluster/log-dump/log- test/e2e/scheduling/BUILD Some of them have a mention of rescheduler in comments but I will remove relevant rescheduler pieces in the code base and will create a PR sometime next week.
Sounds good! Thanks. From the name of the files, it doesn't seem to me that there is any critical piece that we must address ASAP, but I may be wrong.
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.11 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Priority and preemption has been moved to Beta in 1.11. Rescheduler is not completely removed in 1.11, but it is used only to create room for critical DaemonSet pods. Other critical pods in the system must use priority and the scheduler will be responsible for scheduling them.
will this automatically be applied for critical pods you've identified above or will their spec need to be updated?
We have already made changes to run those pods with proper priority. Please check this PR and let us know if we have missed anything. Kube-proxy's priority is also set in a different PR.
Priority and preemption has been promoted to Beta since K8s 1.11.

Discussion:
// The scheduler uses the annotation to find that the pod shouldn't preempt more pods // when it gets to the head of scheduling queue again. // See podEligibleToPreemptOthers() for more information. <del> NominatedNodeAnnotationKey = "NominatedNodeName" </del> <ins> NominatedNodeAnnotationKey = "scheduler.kubernetes.io/nominated-node-name" </ins> ) // Error returns detailed information of why the pod failed to fit on each node