## 3 Research Method
### Dealing with Continuous Evolution of CO Frameworks during the Research  
We have performed the above research from April 2017 till December 2017. After that period, new versions of CO frameworks have of course been continuously released. We have kept the collected information up-to-date as follows. Each time a new version of a CO framework has been released, we reviewed the release notes and change logs of that new version in order to discover feature additions, feature updates and feature deprecations. As a result, new common features have been discovered when a unique feature of a CO framework becomes also supported by another framework; if so, timeline information was also updated. As the article reached completion, we decide to take into account only versions released before 1 July 2018. **4. Genericity: Qualitative Assessment of Common Features**  
We present in this section answers to research questions RQ1-RQ2:  
_RQ1. What are the common features of CO frameworks and what are the different implementation strategies for realizing the common features?_  
_RQ2. How can features be organized in functional (sub)-aspects?_  
We have used the OpenStack user survey as the main inspiration for selecting popular open-source CO frameworks as OpenStack itself is a cloud provider company that is fully rooted in the open-source culture and is a rather neutral with respect to promoting a specific CO framework. Figure 4 gives an overview of the most popular PaaS platforms in OpenStack deployments according to the last two surveys of October 2016 and November 2017. It shows that Kubernetes, OpenShift, Docker Swarm andMesos are the most used container orchestration frameworks for running production-grade services. Note that OpenShift 3.0 [59] has been completely built on top of Kubernetes and Cloud Foundry [60] also provides support for Kubernetes. Moreover, as OpenShift and Cloud Foundry are not pure container orchestration frameworks, but also offer additional PaaS development services, we choose to focus on Docker Swarm, Kubernetes and Mesos for deriving a base of common and unique features. Note that Docker Swarm and Mesos actually cover different frameworks. As such, we compare in total 7 CO frameworks:  
1. Kubernetes [61] supports deploying and managing both service- and job-oriented workloads as sets of containers. Docker Swarm comes with two different distributions:
2. Docker Swarm stand-alone [62] manages a set of nodes as a single virtual host that serves the standard Docker Engine API. Any tool that already communicates with a Docker daemon can thus use this framework to transparently scale to multiple nodes. This framework is minimal but also the most flexible because almost the entire API of the Docker daemon is available. As such it is mostly relevant for platform developers that like to build a custom framework on top of Docker. 3. The newer Docker Swarm integrated mode [63] departs from the stand-alone model by re-positioning Docker as a complete container management platform that consists of several architectural components, one of which is Docker Swarm. 4. Apache Mesos [41; 64] supports fine-grained allocation of resources of a cluster of physical or virtual machines to multiple higher-level _scheduler frameworks_. Such higher-level scheduler frameworks do not only include container orchestration frameworks but also more traditional non-containerized job schedulers such as Hadoop. Figure 4: The two most recent annual OpenStack public user surveys show that Kubernetes, OpenShift, Docker Swarm and Mesos are the most popular container orchestration frameworks in OpenStack deployments. Cloud Foundry has decreased in popularity. Currently, the following three Mesos-based CO frameworks are the most popular:  
* Aurora [65], initially developed by Twitter, supports deploying long-running jobs and services. These workloads can optionally be started inside containers. * Marathon [66] supports deploying groups of applications together and managing their mutual dependencies.