## 3 Research Method
### Container QoS Management  
This aspect covers features of CO frameworks that an application manager must understand in order to efficiently use the resources of a user group while also achieving the intended QoS level of its applications. Supporting high utilization of allocated resources while also maintaining desired QoS levels of applications, during either normal execution or resource contention and failures, is a complex goal. As such, CO frameworks are designed with the following two goals in mind:  
* Resource allocation models have been developed that support QoS differentiation between containers while also allowing for over-subscription of resources to improve server consolidation. * CO frameworks offer various mechanisms to application managers for controlling scheduling decisions that influence the performance of the application. These decisions include the placement of inter-dependent containers and data, and prioritization of containers during resource contention. Note that the offered features do not provide strong SLA guarantees at the level of application-specific metrics (e.g., latency or throughput) but include general mechanisms that can be used to balance the competing goals of improved resource utilization and controllable performance of the application. **Container CPU and memory allocation with support for oversubscription. ** This sub-aspect covers common features of CO frameworks that an application manager must understand how to (i) allocate sufficient resources to a container to achieve its intended performance level, but also to (ii) allow flexible reallocation of idle resources to improve resource utilization. In general, the allocation of computational resources to a container is governed by means of resource allocation policies. Container orchestration frameworks differ in their support for resource allocation policies and also differ in the type of resources that can be limited. In the following, we set out the available support for the different types of resources. _Minimum guarantees and maximum limits for CPU and memory._ Kubernetes and Docker Swarm provide support for minimum guarantees and maximum limits for CPU and memory, while Mesos-based frameworks support minimum guarantees for CPU and maximum limits for both CPU and memory [31]. For example, Kubernetes manages a <request, limit> [213] pair per container and per Pod. A request defines the resource quantity that is always guaranteed to the container (e.g., a requests of 1.5 CPU imply that 1 CPU core and 50% of another CPU core is fully assigned to the container), while a limit specifies the maximum resource quantity that can be used by this container (e.g., a request of 1.5 CPU and a limit of 2 CPU specifies that the container is guaranteed 1.5 CPU cores, but it can take up until 2 full CPU cores if the processing power is not used by other containers). When a CPU limit is exceeded by a container, the container will be throttled [214]. When a memory limit is exceeded, the process using the most memory in the container is killed [215]. Note that when the request is set lower than the limit, the container is guaranteed the request but can opportunistically consume the difference between request and limit if some resources are not being used by other containers. It has been shown in Borg, the predecessor of Kubernetes, that setting requests and limits in the above ways increases resource utilization [42]. _Abstraction of CPU-shares for enforcing CPU guarantees._ Note that Mesos, Aurora, Marathon and Docker Swarm stand-alone rely on CPU-shares of the CFS Scheduler for implementing minimal guarantees. CPU-shares are however difficult to configure because CPU-shares are always defined as weights that are relative to the CPU-shares of other co-located containers: for example, if a new container is started, then the CPU-shares declared by that new container reduce the weights of the already running containers.