Dynamic cluster host interconnectivity based on reachability characteristics

Dynamic cluster host interconnectivity based on reachability characteristics is disclosed. A first host receives a request from a first container executing on the first host to send a communication to a second container on a second host. The first host determines that the first host can communicate with the second host via a layer two communications protocol or that the first host can communicate with the second host only via a layer three communications protocol. The first host identifies, in a host accessibility structure, whether the first host can communicate with the second host via the layer two communications protocol or the layer three communications protocol.

TECHNICAL FIELD

The examples relate generally to communications between hosts in a cluster, and in particular to dynamic cluster host interconnectivity based on reachability characteristics.

BACKGROUND

Hosts in a cluster that execute containers for different tenants, such as different companies, often utilize tunneling protocols, such as virtual local area networks (VLANs), to isolate the traffic of the different tenants.

SUMMARY

The examples disclosed herein implement dynamic cluster host interconnectivity based on reachability characteristics. The examples isolate traffic between containers executing on different hosts, but which are associated with the same tenant, by dynamically selecting a tunneling protocol based on whether the hosts can communicate over a layer two network, or whether the hosts can only communicate over a layer three network.

In one example a method is provided. The method includes receiving, by a first host comprising a processor device, a request from a first container executing on the first host to send a communication to a second container on a second host. The method further includes determining, by the first host, that the first host can communicate with the second host via a layer two communications protocol or that the first host can communicate with the second host only via a layer three communications protocol. The method further includes identifying, in a host accessibility structure, whether the first host can communicate with the second host via the layer two communications protocol or the layer three communications protocol.

In another example a host is provided. The host includes a memory and a processor device coupled to the memory. The processor device is to initiate a first container, receive a request from the first container to send a communication to a second container on a second host, determine that the host can communicate with the second host via a layer two communications protocol or that the host can communicate with the second host only via a layer three communications protocol, and identify, in a host accessibility structure, whether the host can communicate with the second host via the layer two communications protocol or the layer three communications protocol.

In another example a computer program product is provided. The computer program product is stored on a non-transitory computer-readable storage medium and includes instructions to cause a processor device to initiate a first container on a first host. The instructions further cause the processor device to receive a request from the first container to send a communication to a second container on a second host. The instructions further cause the processor device to determine that the first host can communicate with the second host via a layer two communications protocol or that the first host can communicate with the second host only via a layer three communications protocol. The instructions further cause the processor device to identify, in a host accessibility structure, whether the first host can communicate with the second host via the layer two communications protocol or the layer three communications protocol.

DETAILED DESCRIPTION

Any flowcharts discussed herein are necessarily discussed in some sequence for purposes of illustration, but unless otherwise explicitly indicated, the examples are not limited to any particular sequence of steps. The use herein of ordinals in conjunction with an element is solely for distinguishing what might otherwise be similar or identical labels, such as “first host” and “second host,” and does not imply a priority, a type, an importance, or other attribute, unless otherwise stated herein. The term “about” used herein in conjunction with a numeric value means any value that is within a range of ten percent greater than or ten percent less than the numeric value. As used herein and in the claims, the articles “a” and “an” in reference to an element refers to “one or more” of the element unless otherwise explicitly specified.

Hosts in a cluster that execute containers for different tenants, such as different companies, often utilize isolation or tunneling protocols, such as virtual local area networks (VLANs), to isolate the traffic of the different tenants. However, VLANs are implemented over layer two communication protocols, such as a media access control (MAC) layer, and require that all hosts be on the same layer two network. For large scale container cluster environments, such as may be offered by cloud computing service providers, it may be impractical, or at least undesirable, to ensure that all hosts are on the same layer two network. It would be preferable for hosts to be able to be accessible by both a layer two network and a layer three network (e.g., the internet protocol (IP) layer) and still be able to isolate traffic between related containers. In circumstances where a host is only available via a layer three network, layer three isolation or tunneling protocols, such as the virtual extensible local area network (VXLAN) protocol, the Generic Routing Encapsulation (GRE) protocol, the Generic Network Virtualization Encapsulation (GENEVE) protocol, or the Layer 2 Tunneling Protocol (L2TP), may be utilized. However, one host in a container cluster typically does not know whether another host is located on the same layer two network as the one host, or is only accessible over a layer three network, such as the Internet. While an operator may manually configure this information into each host, in the context of large container clusters that involve hundreds or thousands of hosts, and in which containers may be relatively continuously initiated and terminated based on demand, manual configuration of such information may be impractical. It is generally advantageous, where possible, to implement isolation over a layer two communication protocol because isolation/tunneling technologies over a layer three communication protocol can have significant network throughput performance impacts.

The examples disclosed herein implement dynamic cluster host interconnectivity based on reachability characteristics. The examples isolate traffic between containers executing on different hosts, but which are associated with the same tenant, by selecting a tunneling protocol based on whether the hosts can communicate over a layer two network, or whether the hosts can only communicate over a layer three network. In one example, a first container associated with a tenant and which executes on a first host attempts to communicate with a second container associated with the same tenant and which executes on a second host. The first host dynamically determines whether the second host is reachable via a layer two network, such as an Ethernet network, or is only reachable via a layer three network, such as an IP network. The first host may maintain this information in a host accessibility structure. If the second host is reachable via the layer two network, the first host uses a layer two tunneling protocol, such as VLAN, to communicate with the second host. If the second host is only reachable via the layer three network, the first host uses a layer three tunneling protocol, such as VXLAN, to communicate with the second host. Among other advantages, this eliminates a need for an operator to continually manually configure a large number of hosts as new hosts are added to a cluster or removed from a cluster, and ensures that a preferable isolation mechanism is automatically selected to isolate communications between containers that belong to the same ownership group.

FIG. 1is a block diagram of an environment10in which examples may be practiced. The environment10includes a plurality of hosts12-1-12-6(generally, hosts12) that form a cluster of hosts12which are available for executing processes, such as, by way of non-limiting example, containers14-1-14-6(generally, containers14). The initiation and termination of the containers14are managed via a cluster controller16that executes on a host18. The cluster controller16may comprise, by way of non-limiting example, an OpenShift or Kubernetes cluster controller modified to operate in accordance with the examples disclosed herein. The cluster of hosts12executes containers14that are associated with a plurality of different tenants. Tenancy, as used herein, is a categorization mechanism to categorize sets of containers14, such that containers14associated with the same tenant have the same tenant ID, and have a different tenant ID than containers14associated with a different tenant. Communications between containers14associated with the same tenant are generally isolated from containers14associated with another tenant. For example, the hosts12may prevent a container14associated with one tenant from receiving a communication sent by a container14associated with a different tenant. Any characteristic may be used to categorize the containers14via tenancy, such as, by way of non-limiting example, containers14associated with different companies, containers14associated with different departments within the same company, or the like. For purposes of illustration, the examples disclosed herein will use tenancy to refer to different companies, and containers14associated with the same company are assigned the same tenant identifier (ID).

Each host12may comprise, for example, a computing device, or a virtual machine implemented on a computing device. Each host12includes, or shares, a processor device and a memory associated with a computing device. Each host12also has at least one IP address associated therewith. The hosts12-1-12-3,18are communicatively coupled together via a layer two networking mechanism, an Ethernet switch20-1. The phrase “layer two” refers to switching technologies (sometimes referred to as “switches” or “hubs”) that operate by forwarding messages based on addresses that correspond to layer two of the Open Systems Interconnection model (OSI model), sometimes referred to as physical addressing, and that include, for example, forwarding messages based on MAC addresses or other data link layer addressing mechanisms. The hosts12-4-12-6are communicatively coupled together via a layer two (e.g., data link layer) switch20-2. The layer two switches20-1and20-2may be hundreds or thousands of miles apart from one another, and are communicatively coupled together via at least one layer three networking mechanism, in this example a router22. The phrase “layer three” refers to switching technologies (sometimes referred to as “routers”) that operate by forwarding messages based on addresses that correspond to layer three (e.g., network layer) of the OSI model, sometimes referred to as logical addressing, and that include, for example, forwarding messages based on IP addresses or other network layer addressing mechanisms. Although only one router22is illustrated for purposes of simplicity, there may be any number of routers22between the switch20-1and the switch20-2. The router22may be, for example, part of the public Internet, or may be part of a private network owned and managed by an entity that owns and manages the hosts12and18. In some examples, such entity may be a cloud computing service provider.

The containers14are processes implemented via a containerization technology, such as, by way of non-limiting example, CRI-O or Docker. The hosts12, as described in greater detail below, facilitate communications between containers14that have the same tenant ID. While for purposes of illustration and simplicity each host12is illustrated as hosting a single container14, in practice, each host12may host tens, hundreds, or even thousands of different containers14. The containers14on a single host12may be associated with the same tenant, or may be associated with multiple different tenants.

The cluster controller16maintains a host IP table24that contains the IP address of each host12. The cluster controller16may obtain the IP addresses of the hosts12in any of a number of different ways. In one example, an operator26may preconfigure the cluster controller16with the IP address of each host12. In another example, each host12may be preconfigured with the IP address of the host18, and upon initiation of the host12, may register itself with the cluster controller16and provide to the cluster controller16its IP address.

For purposes of illustration, assume that the cluster controller16determines that a new container14associated with a particular tenant having a tenant ID of 2 is to be initiated on one of the hosts12. The mechanism for selecting a particular host12on which to run a container14may be based on any desired algorithm. In one example, the cluster controller16may, based on any desired criteria, select a particular host12, such as the host12-1, and direct the host12-1to initiate a new container14. In other examples, each host12may listen to an event stream that is emitted by the cluster controller16. When the cluster controller16determines that a new container14needs to be initiated, the cluster controller16emits an event that indicates that a new container14needs to be initiated, and one of the hosts12, such as the host12-1, responds to the cluster controller16and indicates that the host12-1is capable of initiating a new container14, and the host12-1then initiates the new container14. Again, solely for purposes of illustration, it will be assumed that the host12-1initiates a new container14-1that has an associated tenant ID (T_ID) of 2. The tenant ID of 2 may be maintained, for example, in metadata that is maintained for each container14by the host12-1.

Similarly, over time, each of the hosts12-2-12-6initiates containers14-2-14-6that are associated with corresponding tenants, as indicated in parentheticals inFIG. 1. The container14-1generates and sends a communication addressed to the container14-2. The host12-1receives a request to send the communication, and determines that the message is destined for the host12-2. The host12-1accesses a host accessibility structure28-1and determines that this is the first attempt to communicate with the host12-2. The host12-1then determines whether the host12-1is accessible via a layer two communications protocol or is accessible only via a layer three communications protocol. The host12-1may make this determination in any suitable manner. In one example, the host12-1sends an address resolution protocol (ARP) message to the host12-2to determine if the host12-2is reachable via a layer two communications protocol. If the ARP message is successful, then the host12-2is accessible via the layer two communications protocol. If the ARP message is not successful, the host12-1may determine that the host12-2is accessible only via a layer three network. Because the host12-2is coupled directly to the same layer two switch20-1as the host12-1, the ARP message is successful, and the host12-1determines that the host12-2is accessible via a layer two communications protocol. The host12-1generates an entry30-1A that identifies that the host12-1can communicate with the host12-2via the layer two communications protocol. The identification may be made via any desired information. In this example, the host12-1identifies the host12-2as being accessible via the layer two communications protocol by identifying in the entry30-1A a layer two tunneling mechanism, a VLAN, that may be used to isolate traffic of different tenants which communicate over a layer two network.

The host12-1determines a VLAN tag for use with the communication from the container14-1. In this example, the host12-1uses the tenant ID of 2 associated with the container14-1. It will be apparent that the VLAN tag could be any desired identifier used to associate containers14with a particular tenant. For example, the VLAN tag may be created as a function of a tenant ID, or the VLAN tag may simply be the tenant ID. The VLAN tag may also be randomly generated and paired with the tenant ID by the cluster controller16when the tenant ID is created.

The host12-1inserts the VLAN tag in an Ethernet packet, and communicates the Ethernet packet to the host12-2. The Ethernet packet is forwarded by the switch20-1to the host12-2. The host12-2receives the Ethernet packet and determines that the VLAN tag of 2 corresponds to the tenant ID of 2. The host12-2determines that the container14-2has a tenant ID of 2, and delivers the Ethernet packet to the container14-2. If the host12-2had determined that the container14-2had a different tenant ID, the host12-2may discard the Ethernet packet and not deliver the Ethernet packet to the container14-2.

Assume that the container14-1generates and sends a second communication addressed to the container14-2. The host12-1receives a request to send the second communication to the container14-2, and determines that the second communication is destined for the host12-2. The host12-1accesses the host accessibility structure28-1and, based on the entry30-1A, determines that it has already been determined that the host12-2is accessible via a layer two tunnel. The host12-1uses the tenant ID of 2 as a VLAN tag in an Ethernet packet, and communicates the Ethernet packet to the host12-2. The Ethernet packet is forwarded by the switch20-1to the host12-2. The host12-2receives the Ethernet packet and determines that the VLAN tag of 2 identifies the tenant ID of 2. The host12-2determines that the container14-2also has a tenant ID of 2, and delivers the Ethernet packet to the container14-2.

Assume that the container14-1generates and sends a third communication addressed to the container14-4. The host12-1receives a request to send the third communication to the container14-4, and determines that the third communication is destined for the host12-4. The host12-1accesses the host accessibility structure28-1and determines that this is the first attempt to communicate with the host12-4. The host12-1then determines whether the host12-4is accessible via a layer two communications protocol or is accessible only via a layer three communications protocol. Because the host12-4is only reachable via the router22, the host12-1determines that the host12-4is accessible only via a layer three communications protocol. The host12-1generates an entry30-2A that identifies that the host12-1can communicate with the host12-4via the layer three communications protocol. The identification may be made via any desired information. In this example, the host12-1identifies the host12-4as being accessible via the layer three communications protocol by identifying in the entry30-2A a layer three tunneling mechanism, a VXLAN, that may be used to isolate traffic of different tenants which communicate over a layer three network.

Depending on the particular layer three tunneling mechanism, the host12-1may first initiate a layer three tunnel between the host12-1and the host12-4. Such initiation may involve any suitable communications handshake between the hosts12-1and12-4required by the particular layer three tunneling mechanism. In this example, no handshake mechanism is necessary for a VXLAN tunnel, and thus the host12-1initially determines a VXLAN tag for use with the communication from the container14-1. In this example, the host12-1uses the tenant ID associated with the container14-1. The phrase “tag” as used herein refers to any labelling mechanism used by a layer two or layer three communications protocol to distinguish traffic between different tenants. For example, in the context of VXLAN, the VXLAN tag may comprise a Network ID, sometimes referred to as a VNID. The host12-1inserts the VXLAN tag of 2 into a communications packet, and communicates the communications packet to the host12-4that follows a communications path through the switch20-1, the router22, and the switch20-2to the host12-4. The communications packet is forwarded by the switch20-2to the host12-4. The host12-4receives the communications packet and determines that the VXLAN tag of 2 corresponds to the tenant ID of 2. The host12-4determines that the container14-4has a tenant ID of 2, and delivers the communications packet to the container14-4. If the host12-4had determined that the container14-4was associated with a different tenant ID, the host12-4may discard the communications packet and not deliver the communications packet to the container14-4.

In this manner, the hosts12can automatically, dynamically, and without human involvement, determine a best tunneling/isolation mechanism between hosts12whether the hosts12are on a same layer two network, or are reachable only via a layer three network.

FIG. 2is a flowchart of a method for dynamic cluster host interconnectivity based on reachability characteristics according to one example.FIG. 2will be discussed in conjunction withFIG. 1. The host12-1receives a request from the container14-1executing on the host12-1to send a communication to the container14-2on the host12-2(FIG. 2, block100). The host12-1determines that the host12-1can communicate with the host12-2via a layer two communications protocol or that the host12-2can communicate with the host12-2only via a layer three communications protocol (FIG. 2, block102). The host12-1identifies, in the host accessibility structure28-1, whether the host12-1can communicate with the host12-2via the layer two communications protocol or the layer three communications protocol (FIG. 2, block104).

FIG. 3is a block diagram of the environment10illustrated inFIG. 1, showing additional aspects of the examples. A container14-7executing on the host12-1sends a communication addressed to a container14-8executing on the host12-2. The host12-1receives the request to send the communication to the container14-8, and determines that the container14-8executes on the host12-2. The host12-1accesses the host accessibility structure28-1and, based on the entry30-1A, determines that it has already been determined that the host12-2is accessible via a layer two tunnel. The host12-1uses the tenant ID of 5 as a VLAN tag in an Ethernet packet, and communicates the Ethernet packet to the host12-2. The Ethernet packet is forwarded by the switch20-1to the host12-2. The host12-2receives the Ethernet packet and determines that the VLAN tag of 5 identifies the tenant ID of 5. The host12-2determines that the container14-8has a tenant ID of 5, and delivers the Ethernet packet to the container14-8.

The container14-7sends a second communication addressed to a container14-9executing on the host12-4. The host12-1receives the request to send the communication to the container14-9, and determines that the container14-9executes on the host12-4. The host12-1accesses the host accessibility structure28-1and, based on the entry30-2A, determines that it has already been determined that the host12-4is accessible via a layer three tunnel.

The host12-1determines a VXLAN tag for use with the communication from the container14-1. In this example, the host12-1uses the tenant ID of 5 associated with the container14-7. The host12-1inserts the VXLAN tag of 5 into a communications packet, and communicates the communications packet to the host12-4that follows a communications path through the switch20-1, the router22, and the switch20-2to the host12-4. The communications packet is forwarded by the switch20-2to the host12-4. The host12-4receives the communications packet and determines that the VXLAN tag of 5 corresponds to the tenant ID of 5. The host12-4determines that the container14-9has a tenant ID of 5, and delivers the communications packet to the container14-9.

In some examples, if a host12is accessible only via a layer three communications protocol, a host12may automatically initiate an Internet Protocol Security (IPsec) tunnel in conjunction with the VXLAN tunnel so that the communications between the containers14are encrypted. This may be particularly suitable where the router22is part of a public network, such as the internet.

FIG. 4is a simplified block diagram of the environment10illustrated inFIG. 1. The host12-1includes a memory32and a processor device34coupled to the memory32. The host12-1may comprise a computing device, sometimes referred to as a “bare metal” computing device, or, in other examples may comprise a virtual machine that is implemented on a computing device. In the latter examples, the virtual machine may have use of the processor device34for intervals of time, and other virtual machines implemented on the computing device may have use of the processor device34for other intervals of time. In some examples, the functionality discussed above with regard to the host12-1may be implemented, at least in part, via software instructions that program the processor device34to implement the functionality discussed above. In such examples, functionality implemented by the host12-1may also be attributed to the processor device34since the processor device34is a component of the host12-1. The processor device34is to initiate the container14-1. The processor device34is further to receive a request from the container14-1to send a communication to the container14-4on the second host12-2. The processor device34is to determine that the host12-1can communicate with the host12-4via a layer two communications protocol or that the host12-1can communicate with the host12-4only via a layer three communications protocol. The processor device34is to identify, in the host accessibility structure28-1, that the host12-1can communicate with the host12-4only via the layer three communications protocol.

FIG. 5is a block diagram of the host12-1suitable for implementing examples according to one example. The host12-1may comprise any computing or electronic device capable of including firmware, hardware, and/or executing software instructions to implement the functionality described herein, such as a computer server, a desktop computing device, a laptop computing device, a smartphone, a computing tablet, or the like. As discussed above, in some examples, the host12-1may also comprise a virtual machine. The host12-1includes the processor device34, the memory32, and a system bus36. The system bus36provides an interface for system components including, but not limited to, the memory32and the processor device34. The processor device34can be any commercially available or proprietary processor.

The system bus36may be any of several types of bus structures that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and/or a local bus using any of a variety of commercially available bus architectures. The memory32may include non-volatile memory38(e.g., read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.), and volatile memory40(e.g., random-access memory (RAM)). A basic input/output system (BIOS)42may be stored in the non-volatile memory38and can include the basic routines that help to transfer information between elements within the host12-1. The volatile memory40may also include a high-speed RAM, such as static RAM, for caching data.

A number of modules can be stored in the storage device44and in the volatile memory40, including an operating system and one or more program modules, such as the container14-1. All or a portion of the examples may be implemented as a computer program product46stored on a transitory or non-transitory computer-usable or computer-readable storage medium, such as the storage device44, which includes complex programming instructions, such as complex computer-readable program code, to cause the processor device34to carry out the steps described herein. Thus, the computer-readable program code can comprise software instructions for implementing the functionality of the examples described herein when executed on the processor device34.

The operator26(FIG. 1), may also be able to enter one or more configuration commands through a keyboard (not illustrated), a pointing device such as a mouse (not illustrated), or a touch-sensitive surface such as a display device (not illustrated). Such input devices may be connected to the processor device34through an input device interface48that is coupled to the system bus36but can be connected by other interfaces such as a parallel port, an Institute of Electrical and Electronic Engineers (IEEE) 1394 serial port, a Universal Serial Bus (USB) port, an IR interface, and the like. The host12-1may also include a communications interface50suitable for communicating with the switch20-1as appropriate or desired.