## 3 Research Method  
This section presents how we have been working towards studying the genericity, maturity, and stability of the CO frameworks. The reader can skip this section if she or he is not interested in these methodological aspects of our work.  
Before starting the research for this article, we have already acquired plenty of experience with container orchestration frameworks in the context of the DeCOMAdS research project [48] that aims to design advanced deployment and configuration middleware for adaptive multi-tenant SaaS applications. At the beginning of this project, we performed a technical SWOT (strengths, weaknesses, opportunities and threats) analysis of containers and container orchestration framework that helps a SaaS provider to make a cost-benefit analysis to move their applications to a container orchestration framework [2]. Subsequently, we have also compared the performance of Docker Swarm and Kubernetes for NoSQL databases [49] and we have built a tool for comparing different auto-scalers for container-orchestrated services in Kubernetes [50].  
We have processed the documentation of the CO frameworks on GitHub because this platform has been used to manage the editing of the documentation as well as the versioning of documentation. To manage the documentation of different versions of a CO framework, git tags are typically used. These git tags allow us to dynamically browse through different versions of the documentation. This was essential for us in order to discover the addition and removal of features across versions (see Section 3.4). Only DC/OS doesn't use git tags, therefore, we have used the official website of DC/OS.  
Figure 3 presents a workflow diagram of how we have extracted all relevant data from the documentation of the CO frameworks on GitHub. This data involves information about the three aforementioned software quality properties of interest. After consulting with experts in mining software repositories, it became clear that no useful or robust tools for automating one or more of these 7 steps existed. Research in the area of processing software documentation [51; 52] is in its infancy. We therefore have manually executed these steps using simple repetitive strategies and pseudo algorithms in order to reduce human error extract the relevant data from GitHub.  
Figure 3: Overview of the method for collecting data from GitHub with respect to genericity, maturity and stability. The top of the diagram shows the 7 successive steps performed. Each step produced specific data that is related to one or more software quality properties. Each of these software qualities are presented by a colored bar. For each step, the extracted data items are specified directly below.  
The following six subsections explain our method for (1) the qualitative assessment and (2) the quantitative analysis with respect to the genericity property--using the data collected in the blue bar of Figure 3, (3) the qualitative assessment of maturity using the data in the green bar, (4) the quality assessment of stability using the data in the red bar. We also explain (5) how we have gathered feedback from industry to improve the coherency and correctness of the collected data and (6) how we have dealt with the continuous evolution of the CO frameworks during the course of the research work.