## 3 Research Method
### Container Networking  
This aspect corresponds with features of CO frameworks that a cluster administrator must understand in order to customize how containers are networked, load balanced and discovered. **Services networking. ** A container exposes a certain service at a well-defined container port. In order to support availability and fault-tolerance, multiple replicas of the container have to be started across multiple nodes and health checked. In order to support connectivity to such container-based, replicated services the following elements are necessary: (i) a stable service name or IP address that is unique to this service irrespective of the state of the pool of containers of that service, (ii) a network with a unique network address for each container, and (iii) a service proxy that enables to lookup service network addresses and translate them to container replica network addresses; the service proxy may also encompass a load balancer to spread the workload of a service across the different replica's. There are three different approaches to enable these three elements of services networking. We consider them as parent features that can be decomposed into a number of child features:  
_Routing mesh for global service ports._ Here, (i) every service is identified by means of a unique port that is opened at each node of the cluster where a container replica runs, (ii) a container is thus addressed using the IP address of its local cluster node and the unique service port, (iii) at one or more nodes of the cluster a load balancer serves requests to a service port by forwarding the requests to the cluster nodes where the containers of that service are running. Load balancers (LBs) can be classified according to the following sub-features:  
1. Whether the LB is _automatically distributed_ on every node of the cluster vs. _centrally installed_ on a few nodes _by the cluster administrator_. In the latter case, sending a request to a service port requires a multi-hop network routing to an instance of the LB. 2. Whether the LB supports _Layer 4_ (i.e., TCP/UDP) vs _Layer 7_ (i.e., HTTPS) load balancing. Layer 7 load balancing allows implementing application-specific load-balancing policies. 3. Whether the Layer 4 LB implementation is _based on the ipos load balancing module of the Linux kernel._ This ipvs module is known as a highly-performing load balancer implementation [113]. 4. Whether containers can run in _bridged_ or in _virtual network_ mode. In the former mode containers can only be accessed via a host port of the local node; the host port is mapped to the container port via a local virtual bridge. In the latter case, remote network connections to a container can be served. Docker Swarm integrated mode [114], Kubernetes [115], Marathon [116] and DC/OS [117; 118] provide support for a routing mesh. An overview of how these CO frameworks can be classified according to the above sub-features is provided in Table 2 and a detailed account is provided in (Section 4.3.1 of the technical report [31]). _Virtual IP network for containers_. Here, (i) each service is either identified by means of a stable DNS name or a stable service IP address, (ii) containers run in virtual network mode, i.e., each container has a unique IP address to which can be remotely connected by means of a virtual network; this virtual network is supported by overlay network software that preferably supports IPv6 network addresses in order to allow for a massive amount of containers in a single cluster, (iii) Service IPs are load balanced by an automatically distributed Layer 4, ipvs-based load balancer, (iv) DNS names are served by an internal DNS service that is automatically installed at one or multiple nodes of the cluster. For load-balanced services with a Service IP, the DNS service resolves to the Service IP by default, otherwise, the DNS services resolves to the list of IP addresses of the containers behind that service.