Project: Koordinator (Orchestration & Management / Scheduling & Orchestration)
File: 20220707-rename-extended-resource.md

title: Refactor extended resource for overcommitment authors: "@zwzhang0107" reviews: "@jasonliu747" "@stormgbs" "@hormes" - - - - - - Using `kubernetes.io` as the namespace for koordinator extended resources, so that `koord-batch` resource can be over-committed and `koord-batch-cpu` cores can be requested as non-integer with better readability. Koordinator defines extended resource for co-location such as `batch-cpu` and `batch-memory`. Due to the limitation of api-server, resources outside the `*kubernetes.io` namespace must be integers and cannot be over-committed (see https://github.com/kubernetes/kubernetes/pull/48922). This proposal suggests using `koordinator.kubernetes.io` as the namespace of koordinator extended resources. Refactor the extended resource of koordinator to use `kubernetes.io` as namespace. Remove the constraints of api-server for extended resource (https://github.com/kubernetes/kubernetes/issues/110536) Co-located pods request resources as the following format to achieve resource overcommitment (request < limit). ```yaml resources: requests: kubernetes.io/batch-cpu: 1500 # 1.5 core kubernetes.io/batch-memory: 1Gi limites: kubernetes.io/batch-cpu: 3000 # 3 core kubernetes.io/batch-memory: 2Gi ``` Defines new formats of extended resources as `kubernetes.io/batch-cpu` and `kubernetes.io/batch-memory` Koord-manager updates new extended resources to `Node.Status`. Webhook of `ColocationProfile` injects new extended resources to `Pod`. Considering pods with the old format may continuously exist for a period of time, koord-scheduler will provide a filter plugin of resource allocation, summarizing the two kinds of batch resource request as allocated resource of node. Here are the pros and cons for alternative plans. Using annotations to express batch resource requests and limits Using `koordinato.sh/` namespace for koordinator batch resources. Using `koordinato.sh/` for resource requests and annotations for resource limits. | Alternative | Pros | Cons | ||-|--| | annotations["batch-cpu"] | resource overcommitment; | incompatible with k8s scheduler | | extended resource `koordinator.sh/batch-cpu` | compatible with k8s scheduler | resource overcommitment is not allowed; | | setting requests as extended resource `koordinator.sh/batch-cpu` and limits in pod annotations | barely works with ugly design | bad expression ; confusing using habits | Extended resources with old formats (koordinator.sh) on node needs to be removed manually. Old format of extended resource will be marked as `deprecated`, koordinator will not support in next few versions. So please update the pod with old format as soon as possible. Although new pods will be injected with new formats of extended resource, there may be some unscheduled pods with old formats of extended resource. If all nodes with old format happen to run out, these pods needs to be deleted manually for resubmissions. 2022-07-07: Initialize proposal for review 2022-08-10: Fix the explanation that batch-cpu still need to be filled as milli-core, since the `scheduler` and `kubelet` will round down by `Value()` for extended-resource