## 3 Research Method
### Application Configuration and Deployment
_Global containers._ For some applications or framework support services, it is necessary that a particular container image is running at every node of the cluster. This concept is supported in Docker Swarm integrated mode [165] and Kubernetes [166] as respectively global services and daemon sets. _Composite applications._ Docker Swarm integrated mode and Marathon provide support for deploying multiple tiers of a distributed application in such an order that the dependencies between the tiers are respected. In Docker Swarm integrated mode [167], different service configurations and their mutual dependencies can be specified as part of a ComposeV3 file [70]. The docker stack deploy command takes as input such ComposeV3 file and deploys all the specified services together as one group while respecting the interdependencies between the services. Marathon supports a similar concept called Application groups [168]. Kubernetes does not natively support a similar concept, but several tools such as Helm [169] and Kompose [170] can be employed. **Persistent volumes. ** In all container orchestration frameworks, containers are stateless; when a container dies, any internal state is lost. Therefore, so-called persistent volumes, which store the persistent state, can be attached to containers. Persistent volumes can be implemented by various mechanisms: services for attaching block storage devices to virtual machines such as Cinder, distributed file frameworks such as NFS, cloud services such as Google Persistent Disk, or local volumes that reserve a subset of the local disk resources. Persistent volume mechanisms can be categorized according to the following 9 features:  
_Local volumes_ that are comprised of disk resources of a container's local host node are supported by all CO frameworks. _Automatic (re)scheduling of containers to local volumes._ Containers that are configured to use a specific local volume are automatically (re)scheduled to the node where that local volume resides. _Shareable volumes between containers_ can be used as an asynchronous data communication channel between containers. Note however, that in general, not all types of persistent volumes support sharing. _External persistent volumes_ are supported by all CO frameworks. Such external volumes support managing data sizes that exceed a node's disk capacity and also allow for state recovery in case of node failures. _Volume plugin architecture._ All CO frameworks except Aurora support a unified interface for different volume implementations. Overall there are two different architecture that are adopted by multiple CO frameworks: the Docker Engine plugin framework and the CSI-based plugins:  
* _Support for Docker volume plugin architecture._ The Docker Engine plugin framework [171], which offers a unified interface between the container runtime and various volume plugins, is adopted by Mesos [172], Marathon [173] and DC/OS [174] in order to support external persistent volumes. In Mesos-based frameworks, Docker volume plugins must be integrated via a separate Mesos module, named dvdi [175], which requires writing plugin-specific glue code. As such a limited number of Docker volume plugins are currently supported in Mesos. * _Support for the Common Storage Interface (CSI) specification._ The Common Storage Interface specification [176] aims to provide a common interface for volume plugins so that each volume plugin needs to written only once and can be used in any container orchestration framework. The specification also supports run-time installation of volume plugins. Typically, CSI can be implemented in any CO framework as a normal volume plugin, which itself is capable interacting with multiple external CSI-based volume plugins. Currently, CSI has been adopted by Kubernetes [177], Mesos [178] and DC/OS [179]. Support for run-time installation of volume plugins has been supported by the Docker Engine plugin framework [180] since Docker engine v1.12 and therefore also supported by Docker Swarm, Mesos, Marathon and DC/OS.