Project: Katalyst (Orchestration & Management / Scheduling & Orchestration)
File: 20221018-qos-resource-manager.md

title: QoS Resource Manager authors: "csfldf" reviewers: "waynepeking348" "xuchen-xiaoying" creation-date: 2022-10-18 last-updated: 2023-02-22 status: implemented see-also: "https://github.com/kubewharf/enhanced-k8s/tree/70544afae9af1c7e1d129069e80f4eceb7d039f5/docs/design/qos-resource-manager" <!-- toc --> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - <!-- /toc --> Despite the fact that the CPU Manager and Memory Manager in kubelet can allocate `cpuset.cpus` and `cpuset.mems` with numa affinity, they have some restrictions and are difficult to customize since all policies of them share the same checkpoint. For instance, only pods with Guaranteed QoS class can be allocated with exclusive `cpuset.cpus` and `cpuset.mems`, but in each individual production environment, Kubernetes original QoS classes may be not flexible enough to depict workloads with different QoS requirements. Besides, those allocation logic works in a static way cause it only counts on numerical values of each resource, without considering the running state of each node. Finally, the current implementation is not pluggable, and if new policies or additional resource managers (like disk quota or network bandwidth) are needed, we have to make changes for kubelet source codes and update kubelet for clusters and that would be costly. Thus, we propose the `QoS Resource Manager` (abbreviated to `QRM` later in the article) as a new component in kubelet ecosystem. It extends the ability of resource allocation in admission phase, and enable dynamic resource allocation adjustment for pods with better flexibility. QRM works in a similar way like the Device Manager, and resource allocation logic will be implemented in external plugins. It will then periodically collect latest resource allocation results, along with real-time node running states, and assemble them as parameters to update through standard CRI interface, like `cpuset.cpus`, `cpuset.mems` or any other potential resources needed in the future,. In this way, the allocation and adjustment logic is offloaded to different plugins, and can be customized by user-defined QoS requirements. In addition, we can implement setting and adjustment logic in plugins for cgroup parameters supported in LinuxContainerResources config (eg. `memory.oom_control`, `io.weight`), QRM will set and update those parameters for pods when corresponding plugins registered. Currently, we have already implemented QRM framework and multiple plugins, and they are already running in production to support QoS-Aware and heterogeneous systems. Pluggable:* make it easier to extend additional resource allocation and adjustment (NUMA affinitive NICs, network or memory bandwidth, disk capacity etc.) without modifying kubelet source codes. Adjustable:* dynamically adjust resource allocation and qos-control strategies according to the real-time node running states. Customizable:* all resource plugins can do the customized qos managing by the specific qos definition. Expand or overturn current pod QoS definitions, thus they'll still remain as Guarantee, Burstable and BestEffort. Instead, users should use common annotations to reflect customized QoS types as needed. Replace current implementation of the CPU Manager and Memory Manager, and they will still work as general resource allocation components to match native QoS definitions. QRM is a new component of the kubelet ecosystem proposed to extend resource allocation policies. Besides: QRM is also a hint provider for Topology Manager like Device Manager. The hints are intended to indicate preferred resource affinity, and pind the resources for a container either to a single or a group of NUMA nodes. QRM will not restrict to any native QoS definition; instead, it will pass meta to plugins and plugins should make it with customized policies. To improve resource utilization rate, overselling, either by VPA or by running complementary workloads in one node, is usually used in production

As a result, the resource consumption states will always be in flux, thus static resource allocation (i.e. `cpuset.cpus` or `cpu.cfsquotaus`) is not enough, especially for workloads with high performance requirements. So a real-time, customized and dynamic resource allocation results adjustment mechanism will be needed. The dynamic adjustment for resource allocation results is usually closely tied to the implementation of QoS aware systems and workload characteristics, so it would be better to provide a general framework in kubelet and offload the resource allocation in plugins. QRM works as such a framework. The native CPU/Memory Manager requires that only pods with Guarantee QoS can be allocated with exclusive `cpuset.cpus` or `cpuset.mems`, but this abstraction lacks flexibility. For instance, in a hybrid cluster, `offline ETL workloads`, `latency-sensitive web services` and`storage services` may run in one same node. In this case, there may be three kinds of `cpuset.cpus` pools, one for offline workloads with shared `cpuset.cpus`, one for web services with shared or exclusive `cpuset.cpus`, and one for storage services with exclusive `cpuset.cpus`. And the same logic may be required for `cpuset.mems` allocation. In other words, we need a `role-based` or `fine-grained` QoS classification and corresponding resource allocation logic, which is uneasy to implement in general CPU/Memory Manager, but can be implemented in user-developed plugins. Consider a node that has multiple NUMA nodes and network interfaces, and pods scheduled to this node want to stick to single NUMA and only use network interface that is affiliated with the NUMA. In this case, multiple pods need to be allocated with the same network device. Although the Device Manager and device plugins can be used, it only allows for `exclusive mode`, ie, device can only be allocated to one certain container. A possible workaround is to allocate `fake device` and set its amount to a large enough value, but `fake device` is kind of weird for end users to request as a resource. With the help of QRM, we also express this `implicit` allocation requirements in annotations and make the customized plugins support it. To increase the UX, the number of new kubelet flags was minimized to a minimum. The minimum set of kubelet flags, which is necessary to configure the QoS Resource Manger, is presented in this section. As shown in the figure above, QRM works as both a plugin handler added to kubelet plugin manager, and a hint provider for Topology Manager. As a plugin handler, QRM is responsible for plugin registration of new plugins, and brings the resource allocation results into effect through standard CRI interface. Detailed strategy is actually implemented in plugins, including NUMA affinity calculation, resources allocation and dynamic adjustment for resources or cgroup parameter control knobs (eg. `memory.oom_control`, `io.weight`, ...). Based on dynamic plugin discovery functionality in kubelet, plugins will register to QRM automatically and make effects during pod lifecycle. As a hint provider, QRM will get preferred NUMA affinity hints of resources with corresponding plugins registered for a container. The figure below illustrates the workflow of QRM, including two major processes the synchronous workflow of pod admission and resource allocation and the asynchronous workflow of periodical resource adjustment Once kubelet requests a pod admission, for each container in a pod, Topology Manager will query QRM (along with other hind providers if needed) about the preferred NUMA affinity for each resource that container requests. QRM will call `GetTopologyHints()` of the resource plugin and get the preferred NUMA affinity for this resource, and return hints for each resource to Topology Manager. Topology Manager then will figure out which NUMA node or a group of NUMA nodes are the best fit for resources/devices affinity aligned allocation after merging hints from all hint

After getting the best fit, Topology Manager will call `Allocate()` of all hint providers for the container. When `Allocate()` is called, QRM will assemble a ResourceRequest for each resource. ResourceRequest contains pod and container metadata, pod `role` and resource `QoS type`, the requested resource name and request/limit quantities and the best hint got in the previous step. To be noted, pod role and resource QoS type are newly defined, and are extracted by `kubernetes.io/pod-role` and `kubernetes.io/resource-type` from pod annotations. They are used to uniquely identify the pod QoS type which will influence the allocation results. QRM will then call `Allocate()` of resource plugin by the ResourceRequest and get the ResourceAllocationResponse. The ResourceAllocationResponse contains properties AllocatedQuantity, AllocationResult, OciPropertyName, Envs, Annotations. etc. A possible ResourceAllocationResponse example for QRM CPU plugin would be like: ``` AllocatedQuantity: 4 OciPropertyName: CpusetCpus (matching with property name in LinuxContainerResources) AllocatationResult: "0-1,8-9" (matching with cgroup cpuset.cpus format) ``` After successfully getting the ResourceAllocationResponse, QRM will cache the allocation result in the pod resources checkpoint, and make the checkpoint persistent by writing checkpoint file. In PreCreateContainer phase, kubeGenericRuntimeManager indirectly will call `GetResourceRunContainerOptions()` of QRM. And the allocation results for the container cached in the pod resources checkpoint will be populated into of CRI api by reflecting mechanism. The completed LinuxContainerResources config will be embedded in the , and be passed as a parameter of API. In this way, the resource allocation results or cgroup parameter control knobs generated by QRM will be taken into the runtime. The overall calculation is performed for all containers in the pod, and if none of containers is rejected, pod finally becomes admitted and deployed. Dynamic resource adjustment is provided as an enhancement for static resource allocation, and is needed in some cases (e.g. QoS aware resource adjustment for resource utilization improvement mentioned in user story 1 above). To support this case, QRM invokes `reconcileState()` periodically, and afterwards calls`GetResourcesAllocation()` of every registered plugin to get latest resource allocation results. QRM will then update pod resources checkpoint and call to update cgroup configs for containers by latest resource allocation results. Pod resources checkpoint is a cache for QRM to keep track of the resource allocation results. It is made by registered resource plugins for all active containers and their resources requests. The structure is shown below: ``` type ResourceAllocation map[string]*ResourceAllocationInfo // Keyed by resourceName type ContainerResources map[string]ResourceAllocation // Keyed by containerName type PodResources map[string]ContainerResources // Keyed by podUID type podResourcesChk struct { sync.RWMutex resources PodResources // Keyed by podUID } type ResourceAllocationInfo struct { OciPropertyName string `protobuf:"bytes,1,opt,name=ocipropertyname,json=ociPropertyName,proto3" json:"ocipropertyname,omitempty"` IsNodeResource bool `protobuf:"varint,2,opt,name=isnoderesource,json=isNodeResource,proto3" json:"isnoderesource,omitempty"` IsScalarResource bool `protobuf:"varint,3,opt,name=isscalarresource,json=isScalarResource,proto3" json:"isscalarresource,omitempty"` // only for resources with true value of IsScalarResource AllocatedQuantity float64 `protobuf:"fixed64,4,opt,name=allocatedquantity,json=allocatedQuantity,proto3" json:"allocatedquantity,omitempty"` AllocatationResult string `protobuf:"bytes,5,opt,name=allocatationresult,json=allocatationResult,proto3" json:"allocatationresult,omitempty"` Envs map[string]string `protobuf:"bytes,6,rep,name=envs,proto3" json:"envs,omitempty" protobufkey:"bytes,1,opt,name=key,proto3" protobufval:"bytes,2,opt,name=value,proto3"` Annotations map[string]string `protobuf:"bytes,7,rep,name=annotations,proto3" json:"annotations,omitempty" protobufkey:"bytes,1,opt,name=key,proto3" protobufval:"bytes,2,opt,name=value,proto3"` ResourceHints *ListOfTopologyHints `protobuf:"bytes,8,opt,name=resourcehints,json=resourceHints,proto3" json:"resourcehints,omitempty"` } ``` PodResources structure is organized as a three-layer map, and it uses pod UID, container name and resource name as keys for each level. ResourceAllocationInfo is stored in the lowest map, and contains the allocation result of a specific resource for the identified container. ResourceAllocationInfo currently has those properties: ``` OCIPropertyName it's used to identify which property in the config LinuxContainerResources should be populated into. IsScalarResource if it's true, resource allocation results can be quantified, and possibly be used as foundation of scheduling and admitting logic. QRM will compare the requested quantity with the allocated quantity when the active container is re-admitted. IsNodeResource if this property and IsScalarResource are both true, QRM will expose "allocatable" and "capacity" quantities of the resource to kubelet node status setter, and finally set to node

For instance, quantified resources have been covered in kubelet node status setter (eg. cpu, memory, ..), it's unnecessary to set quantity of them to node status again, so we should set IsNodeResource as false. Otherwise, we should set IsNodeResource as true for those extended quantified resources, and set quantity of them to node status. AllocatedQuantity it represents resource allocated quantity for the container, and it's only for resources with IsScalarResource set as true. AllocatationResult it represents resource allocation result for the container, and must be a valid value of the property in LinuxContainerResources that OciPropertyName indicates. For example, if OciPropertyName is CpusetCpus, the AllocatationResult should be like "0-1,8-9" (a valid value for cgroup cpuset.cpus). Envs environmental variables that the resource plugin returns and should be set in the container. Annotations annotations that the resource plugin returns and should be taken to runtime. ResourceHints it's the preferred NUMA affinity matching with the AllocatationResult. It will be used when kubelet restarts, and active containers with allocated resources are re-admitted. ``` Assume that we have a machine with CPU Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GH (80 CPUs, 0-39 in NUMA node0, 40-79 in NUMA node1). We divide CPUs in this machine into two pools, one for common online micro-service (eg. web service), and the other for common offline workloads (eg. ETL or video transcoding tasks). And we deploy a QoS aware system to adjust the size of those two pools according to the performance metrics. QRM CPU Plugin starts to work: Initialize those two `cpuset.cpus` pools suppose `0-37,40-77` for online pool, and `38-39,78-79` for offline pool by default QRM CPU Plugin is discovered by kubelet plugin manager dynamically and register to QRM Suppose there comes a pod1 with one container requesting 4 CPUs, and its pod role is online-micro_service, meaning that it should be placed in the online pool. In pod1 admission phase, QRM calls `Allocate()` of QRM CPU plugin. And plugin identifies that the container is for online `cpuset.cpus` pool, so it returns `ResourceAllocationResponse` like below: ``` { "pod_uid": "054a696f-d176-488d-b228-d6046faaf67c", "pod_namespace": "default", "pod_name": "pod1", "container_name": "container0", "container_type": "MAIN", "container_index": 0, "podrole": "online-microservice", "resource_name": "cpu", "allocatation_result": { "resource_allocation": { "cpu": { "ocipropertyname": "CpusetCpus", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 76, "allocatation_result": "0-37,40-77", "resource_hints": [] // no NUMA preference } } } } ``` QRM caches the allocation result in the pod resources checkpoint. In the PreCreateContainer phase, the kubeGenericRuntimeManager calls `GetResourceRunContainerOptions()` of QRM and gets a `LinuxContainerResources` config like below ``` { "cpusetCpus": "0-37,40-77", } ``` pod1 starts successfully with the `cpuset.cpus` allocated. Suppose there comes a pod2 with one container requesting 2 CPUs, and its pod role is ETL, meaning that it should be placed in the offline pool. In pod2 admission phase, QRM call `Allocate()` of QRM CPU plugin. And plugin identifies the container identifies that the container is for offline `cpuset.cpus` pool, so it returns `ResourceAllocationResponse` like below: ``` { "pod_uid": "fe8d9c25-6fb4-4983-908f-08e39ebeafe7", "pod_namespace": "default", "pod_name": "pod2", "container_name": "container0", "container_type": "MAIN", "container_index": 0, "pod_role": "ETL", "resource_name": "cpu", "allocatation_result": { "resource_allocation": { "cpu": { "ocipropertyname": "CpusetCpus", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 4, "allocatation_result": "38-39,78-79", "resource_hints": [] // no NUMA preference } } } } ``` QRM cache the allocation result in the pod resources checkpoint. In the PreCreateContainer phase, the kubeGenericRuntimeManager calls `GetResourceRunContainerOptions()` of QRM and gets a `LinuxContainerResources` config like below: ``` { "cpusetCpus": "38-39,78-79"", } ``` The pod2 starts successfully with the `cpuset.cpus` allocated. Similar to the pod1, if there comes a pod3 also with online-micro_service role, it will be placed in the online pool

So it will get a `LinuxContainerResources` config like below in the `PreCreateContainer` phase: ``` { "cpusetCpus": "0-37,40-77", } ``` After a period of time, the QoS aware system adjusts the online `cpuset.cpus` pool to `0-13,40-53` and the offline `cpuset.cpus` pool `14-39,54-79`, according the system indicators. QRM invokes `reconcileState()` for latest resource allocation results (like below) from QRM CPU plugin. It then updates the pod resources checkpoint, and calls `UpdateContainerResources()` to update the cgroup resources for containers by latest resource allocation results. ``` { "pod_resources": { "054a696f-d176-488d-b228-d6046faaf67c": { // pod1 "container0": { "cpu": { "ocipropertyname": "CpusetCpus", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 28, "allocatation_result": "0-13,40-53", "resource_hints": [] // no NUMA preference } } }, "fe8d9c25-6fb4-4983-908f-08e39ebeafe7": { // pod2 "container0": { "cpu": { "ocipropertyname": "CpusetCpus", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 52, "allocatation_result": "14-39,54-79", "resource_hints": [] // no NUMA preference } } }, "26731da7-b283-488b-b232-cff611c914e1": { // pod3 "container0": { "cpu": { "ocipropertyname": "CpusetCpus", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 28, "allocatation_result": "0-13,40-53", "resource_hints": [] // no NUMA preference } } }, } } ``` Assume that we have a machine with CPU Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GH (80 CPUs, 0-39 in NUMA node0, 40-79 in NUMA node1; 452 GB memory, 226 GB in NUMA node0, 26 GB in NUMA node1) And we have multiple latency-sensitive services, including storage services, re-caller, retriever and re-ranker services in information retrieval systems. etc. To meet QoS requirements, we should pin them to exclusive NUMA nodes by setting `cpuset.cpus` and `cpuset.mems`. Although the default CPU/Memory Manager have already provided the ability to set `cpuset.cpus` and `cpuset.mems` for containers, current policies are only toward Guaranteed pod QoS class, and can't handle container anti-affinity in NUMA nodes scope with flexibility. QRM CPU/Memory plugins start to work QRM CPU plugin initializes checkpoint with default machine state like below: ``` { "machineState": { "0": { "reserved_cpuset": "0-1", "allocated_cpuset": "", "default_cpuset": "2-39" }, "1": { "reserved_cpuset": "40-41", "allocated_cpuset": "", "default_cpuset": "42-79" } }, "pod_entries": {} } ``` QRM Memory plugin initializes checkpoint with default machine state like below: ``` { "machineState": { "memory": { "0": { "Allocated": "0", "allocatable": "237182648320", "free": "237182648320", "pod_entries": {}, "systemReserved": "524288000", "total": "237706936320" }, "1": { "Allocated": "0", "allocatable": "237282263040", "free": "237282263040", "pod_entries": {}, "systemReserved": "524288000", "total": "237806551040" } } }, "podresourceentries": {}, } ``` All QRM plugins are discovered by kubelet plugin manager dynamically and register to QRM There comes a pod1 with one container requesting 20 CPUs, 40GB memory and its pod role is storage-service. In pod1 admission phase, QRM calls `GetTopologyHints()` of QoS plugins, and gets preferred NUMA affinity hint (10) from both of them. Then QRM calls `Allocate()` of plugins calls Allocate() of QoS CPU plugin gets `ResourceAllocationResponse` like below: ``` { "pod_uid": "bc9e28df-1b5c-4099-8866-6110277184e0", "pod_namespace": "default", "pod_name": "pod1", "container_name": "container0", "container_type": "MAIN", "container_index": 0, "pod_role": "storage-service", "resource_name": "cpu", "allocatation_result": { "resource_allocation": { "cpu": { "ocipropertyname": "CpusetCpus", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 20, "allocatation_result": "2-21", "resource_hints": [ { "nodes": [0], "Preferred": true } ] } } } } ``` calls Allocate() of QoS Memory plugin and gets `ResourceAllocationResponse` like below: ``` { "pod_uid": "bc9e28df-1b5c-4099-8866-6110277184e0", "pod_namespace": "default", "pod_name": "pod1", "container_name": "container0", "container_type": "MAIN", "container_index": 0, "pod_role": "storage-service", "resource_name": "memory", "allocatation_result": { "resource_allocation": { "cpu": { "ocipropertyname": "CpusetMems", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 1, "allocatation_result": "0", "resource_hints": [ { "nodes": [0], "Preferred": true } ] } } } } ``` QRM caches the allocation results in the pod resources checkpoint. In the `PreCreateContainer` phase, the kubeGenericRuntimeManager calls `GetResourceRunContainerOptions()` of QRM and gets a `LinuxContainerResources` config like below: ``` { "cpusetCpus": "2-21", "cpusetMems": "0", } ``` The pod1 starts successfully with the `cpuset.cpus`, `cpuset.mems`

There comes a pod2 with one container requesting 10 CPUs, 20GB memory and its pod role is reranker. Although the quantity of available resources is enough for the pod2, QRM plugins identify that pod1 and pod2 should follow anti-affinity requirement in NUMA nodes scope by pod roles. So `ResourceAllocationResponses` for pod2 from QRM CPU/Memory plugins are like below: ``` { "pod_uid": "6f695526-b07c-4baa-90e3-af1dfed2faf8", "pod_namespace": "default", "pod_name": "pod2", "container_name": "container0", "container_type": "MAIN", "container_index": 0, "pod_role": "storage-service", "resource_name": "cpu", "allocatation_result": { "resource_allocation": { "cpu": { "ocipropertyname": "CpusetCpus", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 10, "allocatation_result": "42-51", "resource_hints": [ { "nodes": [1], "Preferred": true } ] } } } } { "pod_uid": "6f695526-b07c-4baa-90e3-af1dfed2faf8", "pod_namespace": "default", "pod_name": "pod2", "container_name": "container0", "container_type": "MAIN", "container_index": 0, "pod_role": "storage-service", "resource_name": "memory", "allocatation_result": { "resource_allocation": { "cpu": { "ocipropertyname": "CpusetMems", "isnoderesource": false, "isscalarresource": true, "allocated_quantity": 1, "allocatation_result": "1", "resource_hints": [ { "nodes": [1], "Preferred": true } ] } } } } ``` QRM caches the allocation result in the pod resources checkpoint.In the `PreCreateContainer` phase, the `kubeGenericRuntimeManager` calls `GetResourceRunContainerOptions()` of QRM and gets a `LinuxContainerResources` config like below: ``` { "cpusetCpus": "42-51", "cpusetMems": "1", } ``` The pod2 starts successfully with the cpuset.cpus allocated. Assume that we have a machine with CPU Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GH (80 CPUs, 0-39 in NUMA node0, 40-79 in NUMA node1) and 2 Mellanox Technologies MT28841 NICs with speed 25000Mb/s (eth0 is affinitive to NUMA node0, eth1 is affinitive to NUMA node1 and). QRM CPU/Memory/NIC plugins start to work QRM CPU/Memory plugins initialize checkpoint with default machine state (same as example 2) QRM NIC plugin initializes checkpoint with default machine state containing NICs information organized by NUMA affinity: ``` { "machineState": { "0": { "nics": [ { "interface_name": "eth0", "nuam_node": 0, "address": { "ipv6": "fdbd:dc05:3:154::20" } } ] }, "1": { "nics": [ { "interface_name": "eth1", "nuam_node": 1, "address": { "ipv6": "fdbd:dc05:3:155::20" }, "netns": "/var/run/netns/ns1" // must be filled in if the NIC is added to the non-default network namespace } ] }, }, "pod_entries": {} } ``` All QRM plugins are discovered by kubelet plugin manager dynamically and register to QRM We assume that all related resources in NUMA node0 have been allocated and allocatable resources in NUMA node1 are all available. There comes a pod1 with one container requesting 10 CPUs, 20GB memory, hostNetwork true and its pod role is `numa-sharing && cpu-execlusive` (`numa-enhancement` for abbreviation). This role means the pod can be located in the same NUMA node with other pods with the same role, but it should be allocated with exclusive `cpuset.cpus`. In pod1 admission phase, QRM calls `GetTopologyHints()` of QoS plugins, and gets preferred NUMA affinity hint (01) from all of them. Then QRM calls `Allocate()` of plugins calls `Allocate()` of QRM CPU/Memory plugins and gets `ResourceAllocationResponses` similar to example 2 calls `Allocate()` of QRM NIC plugin and gets `ResourceAllocationResponse` like below: ``` { "pod_uid": "c44ba6fd-2ce5-43e0-8d4d-b2224dfaeebd", "pod_namespace": "default", "pod_name": "pod1", "container_name": "container0", "container_type": "MAIN", "container_index": 0, "pod_role": "numa-enhancement", "resource_name": "NIC", "allocatation_result": { "resource_allocation": { "NIC": { "ocipropertyname": "", "isnoderesource": false, "isscalarresource": false, "resource_hints": [ { "nodes": [1], "Preferred": true } ], "envs": { "AFFINITYNICADDR_IPV6": "fdbd:dc05:3:155::20" }, "annotations": { "kubernetes.io/host-netns-path": "/var/run/netns/ns1" } } } } } ``` QRM caches the allocation results in the pod resources checkpoint. In the `PreCreateContainer` phase, the kubeGenericRuntimeManager calls `GetResourceRunContainerOptions()` of QRM and gets a `ResourceRunContainerOptions` like below, and the content will be filled into `ContainerConfig` ``` { "envs": { "AFFINITYNICADDR_IPV6": "fdbd:dc05:3:155::20", }, "annotations": { "kubernetes.io/host-netns-path": "/var/run/netns/ns1" }, "linuxcontainerresources": { "cpusetCpus": "42-51", "cpusetMems": "1", }, } ``` The pod1 starts successfully with the cpuset.cpus, cpuset.mems

In addition, the container process can get IP address of the NIC with NUMA affinity from the environment variable `AFFINITYNICADDR_IPV6` and bind sockets to the address. There comes a pod2 with one container requesting 10 CPUs, 20GB memory, hostNetwork true and its pod role is also `numa-enhancement`. That means the pod2 can be located in the same NUMA node with the pod1 if the available resources satisfy the requirements. Similar to the pod1, the pod2 gets a `ResourceRunContainerOptions` like below in admission phase ``` { "envs": { "AFFINITYNICADDR_IPV6": "fdbd:dc05:3:155::20", }, "annotations": { "kubernetes.io/host-netns-path": "/var/run/netns/ns1" }, "linuxcontainerresources": { "cpusetCpus": "52-61", "cpusetMems": "1", }, } ``` Notice that the pod1 and pod2 share the same NIC. If we implement a device plugin with the device manager to provide NIC information with NUMA affinity, it's exclusive so we can't make multiple pods share the same device ID. A new feature gate flag will be added to enable QRM feature. This feature gate will be disabled by default in the initial releases. Syntax: `--feature-gate=QoSResourceManager=false|true` This flag controls the interval time between QRM invoking `reconcileState()` to adjust allocation results dynamically. If not supplied, its default value is 3s. Syntax: `--qos-resource-manager-reconcile-period=10s|1m` Container Manager will create QRM and register it to Topology Manager as a hint provider. Topology Manager will call out QRM to gather topology hints, and allocate resources with corresponding resource plugins registered during pod admission sequence. kubeGenericRuntimeManager will indirectly call `GetResourceRunContainerOptions()` of QRM, and get `LinuxContainerResources` config populated with allocation results of requested resources during executing `startContainer()`. MachineInfo setter will be extended to indirectly call `GetCapacity()` of QRM to get capacity, allocatable quantity of resources and already-removed resource names from registered resource plugins. In order to get the allocated and allocatable resources managed by QRM and registered resource plugin from pod resources server, we make the container manager implement ResourcesProvider defined below: ``` // ResourcesProvider knows how to provide the resources used by the given container type ResourcesProvider interface { // UpdateAllocatedResources frees any Resources that are bound to terminated pods. UpdateAllocatedResources() // GetResources returns information about the resources assigned to pods and containers in topology aware format GetTopologyAwareResources(pod v1.Pod, container v1.Container) []*podresourcesapi.TopologyAwareResource // GetAllocatableResources returns information about all the resources known to the manager in topology aware format GetTopologyAwareAllocatableResources() []*podresourcesapi.AllocatableTopologyAwareResource } When List() of the pod resources server is called, below calling train will be trigger: (*v1PodResourcesServer).List(...) -> (*containerManagerImpl)).GetTopologyAwareResources(...) -> (*QoSResourceManagerImpl).GetTopologyAwareResources(...) -> (*resourcePluginEndpoint).GetTopologyAwareResources(...) for each registered resource plugin QRM will merge allocated resource responses from resource plugins and the pod resources server will return the merged information to the end user. When GetAllocatableResources() of the pod resources server is called, below calling train will be trigger: (*v1PodResourcesServer).GetAllocatableResources(...) -> (*containerManagerImpl)).GetTopologyAwareAllocatableResources(...) -> (*QoSResourceManagerImpl).GetTopologyAwareAllocatableResources(...) -> (*resourcePluginEndpoint).GetTopologyAwareAllocatableResources(...) for each registered resource plugin QRM will merge allocatable resource responses from resource plugins and the pod resources server will return the merged information to the end user. ``` We will initialize QRM with mock resource plugins registered and cover key points listed below: the plugin mananger can discover the listening resource plugin dynamically and register it into QRM successfully. QRM can return correct LinuxContainerResources config populated with allocation results. Validate allocating resources to containers and getting preferred NUMA affinity hints in QRM through registered resource plugins. Validate that reconcileState() of QRM will update the cgroup configs for containers by latest resource allocation results. GetTopologyAwareAllocatableResources() and GetTopologyAwareResources()of QRM return correct allocatable and allocated to the pod resources server. pod resources checkpoint is stored and resumed normally and its basic operations work as

Feature gate Feature gate name: QoSResourceManager Components depending on the feature gate: kubelet Will enabling / disabling the feature require downtime of the control plane? No Will enabling / disabling the feature require downtime or reprovisioning of a node? (Do not assume Dynamic Kubelet Config feature is enabled). Yes, it uses a feature gate. Yes, the pod admission flow will change if QRM is enabled, it will call the plugins to determine whether the pod is admitted or not. And QRM can't work together with CPU Manager or Memory Manager if there are plugins for CPU/Memory allocation registered to QRM. Yes, it uses a feature gate. QRM uses the state file to track resource allocations. If the state file is not valid, it's better to remove the state file and restart kubelet. (e.g. the state file might become invalid that some pods in it have been removed). But the Manager will reconcile to fix it, so it won't be a big deal. Yes, there is a number of Unit Tests designated for State file validation. It is possible that the State file will have inconsistent data during the rollout, because of kubelet restart, but you can easily fix it by removing State file and restarting kubelet. It should not affect any running workloads. And the Manager will reconcile to fix it, so it won't be a big deal. The pod may fail with the admission error because the plugin fails to allocate resources. You can see the error message under the pod events. Yes. No. QRM data will also be available under Pod Resources API. After pod starts, you have two options to verify if containers work as expected via Pod Resources API, you will need to connect to grpc socket and get information from it. Please see pod resource API doc page for more information. checking the relevant container cgroup under the node. Pod failed to start because of the admission error. This does not seem relevant to this feature. A bunch of metrics will be added to indicate the running states of QRM and the plugins. The detailed metrics name and meaning will be added to this doc in the future. None No No No No No No, the algorithm will run on a single `goroutine` with minimal memory requirements. No. During the enabling and disabling of QRM, you must remove the memory manager state file(/var/lib/kubelet/qosresourcemanager_state), otherwise the kubelet start will fail. You can identify the issue via checking the kubelet log. Not applicable. QRM has been developed and running in our production environment, but still needs a little effort to make an open-source version. No objections exist to implement this KEP. collects topology hints from various hint providers (e.g. CPU Manager or Device Manager) in order to calculate which NUMA nodes can offer a suitable amount of resources for a container. The final decision of Topology Manager is subjected to the topology policy (i.e. best-effort, restricted, single-numa-policy) and possible NUMA affinity for containers. Finally, the Topology Manager determines whether a container in a pod can be deployed to the node or rejected. provides CPU pinning functionality by using cgroups cpuset subsystem, and it also provides topology hints, which indicate CPU core availability at particular NUMA nodes, to Topology Manager. allows device vendors to advertise their device resources such as NIC device or GPU device through their device plugins to kubelet so that the devices can be utilized by containers. Similarly, Device Manager provides topology hints to the Topology Manager. The hints indicate the availability of devices at particular NUMA nodes. the assignment of pre-allocated hugepage resources to