## 3 Research Method
### Application and Cluster Management  
This aspect covers features of CO framework that a cluster administrator or application manager must understand in order to manage various non-functional requirements of respectively the cluster or the containerized applications. These management services rely on the Identity and Access Management functionality (see Section 4.7) in order to support customized instances of their functionality to cluster administrators and application managers. **Creation, management and inspection of cluster and applications. **  
_Command Line Interfaces (CLIs)_ with a well-defined command structure are provided in all CO frameworks in order to offer user-friendly usage of the Master API. _Web UI._ Docker [280], Kubernetes [281] and DC/OS [282] offer beside their HTTP-based Master API and Command-Line Interface (CLI) also a graphical user interface for inspecting and managing the state of all objects that can be managed via the Master API, e.g., nodes, services, containers, replication levels of containers, volumes, user groups, multi-tenant authorization controls etc. Erroneous states such as unhealthy containers or failed nodes can also be inspected. See the technical report [31] for a detailed comparison. _Labels for organizing and selecting subsets of API objects_. The CLI and/or dashboard of Docker, Kubernetes, Mesos and DC/OS also use labels for organizing and selecting subsets of containers, services and nodes according to multiple dimensions (e.g., development vs production environments, front-end vs database tiers). Docker Swarm supports service labels [283] and node labels [284]. In Kubernetes, labels can be attached to various API objects [161] and the Kubernetes CLI and dashboard allows to select objects by their labels. Mesos supports task labels [285], while the DC/OS distribution of Marathon allows one to attach labels to Marathon applications and Mesos tasks [286]. _Inspection of cluster-wide resource usage_. The GUIs and associated CLIs also support inspection of (aggregate statistics of) cluster-wide resource usage in order to inspect global cluster state and health. Docker EE's Universal Control Plane [280] shows CPU and memory resource consumption of nodes. Kubernetes' dashboard [281] shows CPU and memory usage at different levels: cluster-wide, per node, and for each separate pod. Aurora's Observer component [287] enables browser-based access to disk usage metrics per task. DC/OS [282]'s dashboard shows CPU, memory and disk usage at similar levels: cluster-wide, per node, per service. **Monitoring resource usage and health. **  
_Central monitoring of resource usage of services and containers._ Kubernetes [288] supports two kinds of resource metrics pipelines: (i) a core Metrics API [289] that supports monitoring Pods for auto-scaling purposes and (ii) several independent full metrics pipelines [290] of which the most prominent are the Prometheus open-source project with built-in support for Kubernetes and Google Cloud Monitoring. Mesos [291] exposes at every agent an HTTP endpoint with aggregate resource consumption metrics for containers running under that agent. When using marathon-lb [292] for exposing the service of a container, statistics for the network interface of that container [293] can be monitored at the same HTTP endpoint. When using a CNI network [294], network statistics of a container can also be queried. DC/OS supports a central Metrics API [295] for monitoring containers and Marathon applications. This also involves monitoring network usage per container. Central monitoring of resource usage by CO framework componentsBesides monitoring the resource usage of services and containers, Docker Swarm [296], Kubernetes [297], Mesos [298], Aurora [299], Marathon [300] and DC/OS [301] also support central monitoring of (aggregated statistics) of resource usage by CO framework components. Kubernetes [302] and Mesos [303]-based frameworks also support monitoring GPU usage.