## 3 Research Method
### Container QoS Management
Kubernetes [213] and Docker Swarm integrated mode [216], on the other hand, offer higher-level abstractions for expressing minimal guarantees that hide the complexity of CPU-shares. **Allocation of other resources. **  
_Limits for NVIDIA GPU_ are supported by Mesos [217], Aurora [218] and Marathon [219] (and DC/OS [220]). Kubernetes [221] offers partial support for GPU allocation because containers cannot requests fractions of a GPU, only a whole GPU, and a single GPU device can neither be shared between containers. _Limits for disk resources._ Mesos offers support for hard [155] and soft limits for disk usage. Hard limits for disk usage are adopted by Aurora [222], Marathon [219] (and DC/OS). Kubernetes [223] offers support for setting a <request, limit> pair for usage of a node's local root partition (ephemeral storage). **Controlling scheduling behavior by means of placement constraints. ** All CO frameworks allow restricting the placement decision of the default scheduling algorithm by means of various user-specified constraints in order to improve the QoS level of applications. These user-specified constraints support placing inter-dependent application containers and data close or far from each other in the network topology. Different types of constraints are supported:  
_Restrict the set of nodes by evaluating over node labels._ CO frameworks allow restricting the set of nodes on which a specific container can be scheduled by means of evaluating over node labels or attributes. A label is defined as a <key, value> pair. A number of such labels are predefined like the hostname of the node. _Evaluate over custom labels._ Custom labels can also be defined: in Docker Swarm integrated mode [224] and Kubernetes [225], custom labels can be dynamically added or removed, whereas in Marathon [226] and DC/OS [227] custom attributes [228] can only be changed by (re)starting the Mesos agent with the desired list of attributes. _More expressive constraints._ The CO frameworks differ in the expressiveness of the constraints. Docker Swarm integrated mode [229] offers _set-based inclusion operators_ for both label keys and label values. Kubernetes does not only offer the same set-based inclusion operator [230], but also more expressive affinity and anti-affinity constraints [231] and constraints for restricting placement of containers to nodes with specific hardware features such as GPUs [232]. Mesos-based frameworks support SQL-alike queries such as GROUP BY that evenly divides containers across aggregate units such as racks or datacenters [89]. **Controlling preemptive and re-scheduling behavior. ** This sub-aspect covers common features of CO frameworks that an application manager must understand in order to customize the pre-emptive scheduling and rescheduling logic of CO frameworks such that an intended QoS level for a containerized application is achieved during several exceptional conditions: (i) resource contention at the scheduler level, (ii) out-of-resource node conditions, (iii) node failures, (iv) container start failures and (v) unbalanced services of which the containers are not spread across different nodes. _Pre-emptive scheduling._ Kubernetes [233] and Aurora [234] use priorities between containers for killing low-priority containers in case the scheduler cannot find a node with enough available resources for scheduling a new container. _Container eviction when a node runs out of resources._ A fully packed node will likely run out of resources in Kubernetes and Docker Swarm when the maximum resource limits of multiple containers on that node are set higher than their minimum guarantees. After all, the default scheduling algorithm of Kubernetes [235] and Docker Swarm will allocate containers to a node so that, for each resource type, the sum of the containers' minimum guaranteed resources does not exceed the capacity of that node. To handle such out-of-resource conditions, Kubernetes and Aurora distinguish between different QoS classes of containers.