Patent Publication Number: US-10333867-B2

Title: Active-active load-based teaming

Description:
FIELD OF THE INVENTION 
     The various embodiments described in this document relate to the management of network interface controllers. In particular, embodiments relate to detecting connectivity between network interface controllers, transitioning between active-passive and active-active states, and managing network interface controllers in an active-active state of a load balancing configuration. 
     BACKGROUND OF THE INVENTION 
     Hosts in typical datacenters are equipped with two or more network interface controllers (NICs), each of which are connected to separate switches for redundancy. In a load-balancing configuration, such as load-based teaming (LBT), a first NIC for a host is used in active mode to send and receive data, and a second NIC for the host is used in passive mode. In passive mode, the second NIC is configured to only to receive (and not send) data until the first NIC reaches a utilization threshold. Once the utilization threshold is reached, the passive NIC becomes active and the active NIC becomes passive. Because switches update their forwarding tables based on the source addresses (e.g., MAC addresses) of packets they are forwarding, switches connected to NICs operating in passive mode are not aware of the presence of the host on which the passive NIC resides. This configuration, however, can result in flooding and network isolation. 
     Flooding occurs when a packet sent to a destination host reaches a switch connected to a passive NIC of the destination host. As a result of the switch being unaware of the location of the destination host, the switch will flood the data packet along all communication channels to each host connected to the switch. Flooding the data packet to NICs that are not the intended destination results in a reduction of the throughput of each NIC. 
     Network isolation occurs when a first host loses an active link between an active NIC and a first switch. The first host will convert the link between a passive NIC and a second switch from passive to active to allow the first host to send and receive packets. However, a second host with an active link to the first switch and a passive link to the second switch may not be able to reach the first host by sending packets to the first switch and, because the second host has a passive link to the second switch, it is not able to send packets to the first host via the second switch. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention is illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements, and in which: 
         FIG. 1  illustrates, in block diagram form, an exemplary computing environment, including one or more networked hosts configured to manage the states of network interface controllers; 
         FIG. 2  is a flow chart illustrating an exemplary method of detecting connectivity between NICs, transitioning between active-passive and active-active states, and managing NICs in an active-active state of a load balancing configuration; and 
         FIG. 3  is a flow chart illustrating an exemplary method of maintaining network addresses for sending packets in an active-active state of a load balancing configuration. 
     
    
    
     DETAILED DESCRIPTION 
     This document describes embodiments that implement a method of managing network interface controllers (NICs) to determine when NICs within a host operating in active-passive mode can operate in an active-active mode. In the active-active mode, both NICs can send and receive packets. Hosts broadcast a probe message from a first NIC in active mode to determine if the network topology will support operating the NICs in active-active mode. If a second NIC of the same host does not receive the probe message, the host places or maintains the second NIC in an active mode. The first NIC can continue to send probe messages at regular intervals, and if a subsequent probe message is received by the second NIC, the host switches the second NIC to passive mode. When both NICs are in active mode, the host can broadcast the same address on both NICs, thus enabling switches connected to the respective NICs to forward packets without flooding. 
     Furthermore, when a first host is operating in the active-active mode, the first host can monitor its reception of address resolution packets from other hosts. If a first NIC of the first host receives a broadcast with a network address of a second host, the first host monitors the second NIC to determine if the second NIC received the same broadcast. When the second NIC does not receive the same broadcast of the network address of the second host, the first host determines that the second host has lost connection on one of its links. The first host uses this information to reach the second host by using the NIC of the first host that received the broadcast from the second host. Thus, a host losing connection on one NIC will not result in network isolation of that host. 
       FIG. 1  illustrates, in block diagram form, exemplary computing environment  100 , including one or more networked hosts  105  configured to manage the states of network interface controllers. This document may also refer to hosts  105  as nodes, computers, processing devices, and/or servers. In one embodiment, one or more VMs  110  implement a virtualized computer, that can provide computing services such as a network server, remote productivity desktop, or some networking, storage, or security service (e.g., a firewall, webserver, database server, etc.). Although not shown, one or more of VMs  110  may include containerized applications, or be generally referred to as data compute nodes (DCNs) which could include application containers as further described below. 
     Hardware  125  includes one or more processors (CPU(s)), data storage and memory (e.g., “RAM”)  150 , and network interface controllers (“NIC(s)”)  127 . Node  105  uses data storage and memory  150  for storing data, metadata, and programs for execution by the processor(s). Data storage and memory  150  may include one or more of volatile and non-volatile memories, such as Random Access Memory (“RAM”), Read Only Memory (“ROM”), a solid-state drive (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage, such as magnetic storage devices, optical storage devices, etc. The memory may be internal or distributed memory. 
     One or more buses may interconnect the various components of hardware  125 . Additionally, NICs  127  may connect nodes  105 , via a wired or wireless network, with one another. 
     Virtualization software  120  runs on hardware  125  of host server or node (e.g., a physical computer)  105 . Virtualization software  120  manages VMs  110  and physical resources, such as hardware  125 . Additionally, virtualization software  120  maintains virtual-to-physical hardware mappings. For example, virtualization software  120  may manage VM access to a processor, memory, or a network interface within hardware  125 . Additionally, virtualization software  120  may manage access to virtual disks (or portions thereof) and other related files within local storage  150  accessible by VMs  110  residing in one or more nodes  105 . 
     In one embodiment, virtualization software  120  manages one or more virtual network interface controllers (e.g., vNICs  123 ). For example, virtualization software  120  provides each VM  110  with a corresponding vNIC  123 . Virtualization software  120  maps each vNIC  123  to a (physical) NIC  127 , e.g., via a virtual switch (not illustrated). 
     Hosts  105  also include NIC state controller  130 . For example, NIC state controller  130  may be a part of virtualization software  120 . NIC state controller  130  manages the states (active/passive) of NICs  127  as described with reference to  FIGS. 2-3 . While described with reference to computing environment  100 , NIC state controller  130  may also be implemented in other computing environments. For example, NIC state controller  130  may manage the states of NICs  127  as described within this document in a server, computer, or other computing environment that does not include virtual machines. 
     Switches  135  are network devices that receive, process, and forward data from a source to a destination. For example, switch A  135  may be a top of rack switch connected to NIC A  127  and NIC C  127  via connections  141  and  145 , respectively, and switch B  135  may be a top of rack switch connected to NIC B  127  and NIC D  127  via connections  143  and  147 , respectively. Switches A-B  135  forward data traffic to, from, and between hosts  105 . In one embodiment, communication channel  149  between switch A  135  and switch B  135  enables communication between switches  135 . For example, communication channel  149  may be represent a layer  2  (or data link layer) connection or a layer  3  (or network layer) connection. Communication channel  149  is illustrated as a broken line to indicate that it is optional. Thus, in another embodiment, there is no communication channel between switch A  135  and switch B  135 . In one embodiment, host A  105  sends packets to host B  105  by sending the packets via one of switch A  135  and/or switch B  135 . For example, NIC A  127  sends packets across communication channel  141  to switch A  135  and switch A  135  forwards the packets to NIC C  127  across communication channel  145 . As another example, NIC A  127  sends packets across communication channel  141  to switch A  135 , switch A  135  forwards the packets to switch B across communication channel  149 , and switch B  135  forwards the packets to NIC D  127  across communication channel  147 . In yet another example, NIC B  127  sends packets across communication channel  143  to switch B  135  and switch B  135  forwards the packets to NIC D  127  across communication channel  147 . 
       FIG. 2  is a flow chart illustrating exemplary method  200  of detecting connectivity between NICs, transitioning between active-passive and active-active states, and managing NICs in an active-active state of a load balancing configuration. Method  200  is described with reference to host A  105  broadcasting a layer  2  probe message from active NIC A  127  to determine whether NIC B  127  receives the probe message or other packets broadcast from NIC A  127 . This allows host A  105  to gather information regarding the topology of computing environment  100 . Method  200  can be similarly performed using any active NIC  127  within any host  105  in computing environment  100 . 
     At block  205 , NIC state controller  130  uses a first network interface (e.g., NIC A  127 ) of host A  105  connected to switch A  135  transmits a probe message as a layer  2  broadcast. NIC A  127  broadcasts the probe message to switch A  135  across communication channel  141 . For example, the destination address (e.g., destination MAC address) of the probe message is set as FF:FF:FF:FF:FF:FF to broadcast the probe message within the boundary of the local network/layer  2  domain. To avoid conflicts with existing address space, host A  105  selects or uses a designated MAC address that is not in use by any devices or components of computing environment  100  as the source address of the probe message. 
     At block  210 , NIC state controller  130  of host A  105  determines if a second network interface (e.g., NIC B  127 ) of host A  105 , connected to switch B  135 , received the probe message. If host A  105  receives the probe message via NIC B  127 , host A  105  determines that there is a layer  2  connection/path between NIC A  127  and NIC B  127 . For example, communication channel  149  may be a layer  2  connection between switch A  135  and switch B  135 , providing a layer  2  path from NIC A  127  to NIC B  127 . Exemplary layer  2  connections between switch A  135  and switch B  135  include a direct layer  2  link, a layer  2  link via a spine, a link aggregation group (LAG) connection, and a layer  2  fabric connection. As another example, NIC A  127  and NIC B  127  may be connected to the same switch (not illustrated), which would provide a layer  2  connection between NIC A  127  and NIC B  127 . If host A  105  does not receive the probe message via NIC B  127 , host A  105  determines that there is no layer  2  connection/path between NIC A  127  and NIC B  127 . Examples of topology that would not have a layer  2  connection between NIC A  127  and NIC B  127  include communication channel  149  being a multi-channel LAG (MLAG), a Virtual PortChannel (vPC), a representation of each of switch A  135  and switch B  135  having a layer  3  connection to a spine, or another layer  3  connection between switch A  135  and switch B  135 . As another example, a lack of connection (no layer  2  and no layer  3  connection) between switch A  135  and switch B  135  would result in NIC B  127  not receiving the probe message. When host A  105  determines that NIC B  127  did not receive the probe message (e.g., within a threshold period of time), method  200  proceeds to block  215 . When host A  105  determines that NIC B  127  did receive the probe message, method  200  proceeds to block  220 . 
     At block  215 , NIC state controller  130  of host A  105  transmits a network address associated with host A as a layer  2  broadcast from both NIC A  127  and NIC B  127 . In response to determining that NIC B  127  did not receive the probe message, host A  105  determines that there is not an active layer  2  connection (e.g., communication channel  149 ) between switch A  135  and switch B  135 . In such situations, host A  105  can operate both NIC A  127  and NIC B  127  in active mode, allowing both NIC A  127  and NIC B  127  to send and receive data packets. In one embodiment, NICs A-B  127  broadcast a network address for host A  105 . Both switch A  135  and switch B  135  receive the broadcast network address. In one embodiment, NIC A  127  and NIC B  127  broadcast the same network address in a gratuitous address resolution protocol (GARP) packet. For example, NICs A-B  127  may broadcast a link layer address (e.g., MAC address) of a vNIC  123  for a VM  110  running on host A  105 . Switch A  135  and switch B  135  update their corresponding forwarding tables with the source addresses (e.g., link layer address of host A  105 , host B  105 , or VMs  110  broadcast on both NICs A-B  127 ) based on the source link layer address of the GARPs. 
     At block  220 , in response to determining that NIC B  127  did receive the probe message, NIC state controller  130  of host A  105  broadcasts a network address associated with host A  105  from NIC A  127 . In one embodiment, host A determines that there is an active link layer connection (e.g., communication channel  149 ) between switch A  135  and switch B  135  based on NIC B  127  receiving the probe message transmitted from NIC A  127 . In such situations, host A  105  operates NIC A  127  in active mode and NIC B  127  in passive mode, allowing NIC A  127  to send and receive data packets, while only allowing NIC B  127  to receive data packets. In one embodiment, NIC B  127  may have been operating in active mode, and host A  105  switches NIC B  127  from active mode to passive mode. Host A  105  prevents a destination NIC  127  within the same network from receiving multiple copies of the same data packet by operating only one of NIC A  127  and NIC B  127  in active mode. In order to identify NIC A  127  as being active and NIC B  127  as being passive, host A  105  only broadcasts the network address associated with host A  105  from NIC A  127 . NIC B  127  does not broadcast the network address associated with host A  105  while in passive mode. In one embodiment, NIC A  127  broadcasts the network address as a GARP packet. In one embodiment, NIC A  127  broadcasts a network address for host A  105  that is received by switch A  135  and switch B  135 . In one embodiment, NIC A  127  broadcasts one or more of a MAC address of host A  105 , a MAC address of a VM  110  running on host A  105 , or another link layer address as the network address. Switch A  135  receives the broadcast(s) from NIC A  127  and forwards the broadcast(s) to switch B  135 . Switch A  135  and switch B  135  update their corresponding forwarding tables with the source address (e.g., link layer address) of host A  105  or VM  110  based on the source link layer address of the GARP received from NIC A  127 . For example, switch A  135  maps the address broadcast by host A  105  to its port connected to NIC A  127  via communication channel  141  and switch B  135  maps the address broadcast by host A  105  to its port connected to switch A  135  via communication channel  149 . 
     At block  225 , NIC state controller  130  of host A  105  monitors communication channel  141  between NIC A  127  and switch A  135  to determine whether communication channel  141  is operating properly or is down or disconnected. Additionally, host A  105  monitors communication channel  143  between NIC B  127  and switch B  135  to determine whether communication channel  143  is operating properly or is down or disconnected. For the ease of explanation, examples described by this document focus on communication channel  141 . For example, host A  105  detects a lack of a cable connection (e.g., lack of a signal or voltage on a physical port), that data is not being received and/or cannot be sent across communication channel  141 , or otherwise that communication channel  141  has failed. When communication channel  141  is not down or disconnected (i.e., communication channel  141  is operational), method  200  proceeds to block  205  to send out an additional probe message. When host A  105  detects that communication channel  141  is down or disconnected, method  200  proceeds to block  230 . 
     At block  230 , NIC state controller  130  of host A  105  broadcasts the network address associated with host A  105  from NIC B  127  in response to detecting that communication channel  141  is down or disconnected. For example, host A  105  sends a GARP packet to switch B  135  and switch B  135  forwards the GARP packet to any other connected switches  135  and/or hosts  105  within the same layer  2  network. Broadcasting the network address associated with host A  105  from NIC B  127  to switch B  135  notifies one or more other hosts  105  connected to switch B  135  that packets directed to the network address of host A  105  can be delivered to host A  105  via their respective connections to switch B  135 . Additionally, as described with reference to  FIG. 3 , the broadcasting of the network address associated with host A  105  from NIC B  127  but not from NIC A  127  provides an indication to other hosts  105  within the layer  2  network that host A  105  cannot be reached via connections to switch A  135 . 
     Method  200  then proceeds back to block  205 . In one embodiment, NIC state controller  130  sends probe messages at regular intervals. For example, NIC state controller  130  may send a probe message every 30 seconds. In one embodiment, the probe messages are sent periodically to monitor for any changes to the topology of computing environment  100  that occur based on changes to connections, e.g., between hosts  105  and switches  135  or between switches  135 . 
       FIG. 3  is a flow chart illustrating an exemplary method  300  of maintaining network addresses for sending packets in an active-active state of a load balancing configuration. In  FIG. 3 , host A  105  is in active-active mode, where both NIC A  127  and NIC B  127  are configured to send and receive data packets. 
     At block  305 , a first host (e.g., host A  105 ) receives a broadcast of a network address from a second host (e.g., host B  105 ) via a first network interface (e.g., NIC A  127 ). For example, host A  105  receives the broadcast of the network at NIC A  127  from switch A  135  across communication channel  141 . In this example, switch A  135  receives the broadcast of the network address from NIC C  127  across communication channel  145 . This indicates to host A  105  that host B  105  (or a VM  110  running on host B  105 ) is reachable via NIC A  127  and communication channel  141 . 
     At block  310 , NIC state controller  130  of host A  105  determines whether a second network interface (e.g., NIC B  127 ) of host A  105  received the same broadcast of the network address within a threshold period of receiving the broadcast via NIC A  127 . For example, host A  105  monitors for the broadcast of the same network address to arrive at NIC B  127  from switch B  135 . Given that host A  105  is in active-active mode, host A  105  operates under the assumption that other hosts  105  within the network are also in active-active mode. As described with reference to  FIG. 2 , hosts  105  in active-active mode broadcast addresses from both/multiple NICs  127  unless a connection fails. When host A  105  receives the same broadcast of the network address from NIC B  127  within the threshold period, host A  105  detects that host B  105  is operating normally in active-active mode and method  300  proceeds to block  315 . When host A  105  does not receive the same broadcast of the network address via NIC B  127  within the threshold period of receiving the broadcast via NIC A  127 , host A  105  determines that host B  105  has lost a connection and method  300  proceeds to block  320 . 
     At block  315 , NIC state controller  130  of host A  105  updates an address table to send packets to host B  105  using either of NIC A  127  and NIC B  127 . In one embodiment, when host A  105  detects a packet for transmission to the network address associated with host B  105 , because NICs A-B  127  are in active-active mode, NIC state controller  130  of host A  105  selects one of NIC A  127  and NIC B  127  for sending the packet to either switch A  135  or switch B  135 , respectively. For example, host A  105  selects either NIC A  127  or NIC B  127  using a round robin or other load balancing algorithm. As another example, host A  105  selects a NIC  127  based upon a predefined default selection of a NIC  127  when both NICs  127  are an option for sending the packet to host B  105 . 
     In one embodiment, if the network address of host B  105  is accessible from both NIC A  127  and NIC B  127  of host A  105 , NIC state controller  130  of host A  105  uses network I/O control (NIOC) to determine which of NIC A  127  and NIC B  127  to use. In one embodiment, NIOC categorizes traffic into network resource pools, which use resource allocation policies to control bandwidth for various traffic types. For example, if a data packet for sending to host B  105  is of a type that has more bandwidth allocated on NIC A  127  than NIC B  127 , host A prioritizes NIC A  127  as the source for sending the packet to host B  105 . 
     At block  320 , NIC state controller  130  of host A  105  updates an address table to send packets to host B  105  using NIC A  127 . Not receiving the broadcast of the network address at NIC B  127  indicates to host A  105  that communication channel  147  between NIC D  127  in host B  105  and switch B  135  is down or has a loss of connection (or that host B  105  is otherwise inaccessible via NIC B  127 ), and any packets with host B as the destination can only be sent using NIC A  127 . In one embodiment, when host A  105  detects a packet for transmission to the network address associated with host B  105 , host A  105  selects NIC A  127  for sending the packet to switch A  135  across communication channel  143 . 
     It will be apparent from this description that aspects of the inventions may be embodied, at least in part, in software. That is, computer-implemented methods  200  and  300  may be carried out in a computer system or other data processing system, such as nodes  105 , in response to its processor executing sequences of instructions contained in a memory or another non-transitory machine-readable storage medium. The software may further be transmitted or received over a network (not shown) via a network interface. In various embodiments, hardwired circuitry may be used in combination with the software instructions to implement the present embodiments. It will also be appreciated that additional components, not shown, may also be part of nodes  105 , and, in some embodiments, fewer components than that shown in  FIG. 1  may also be used in nodes  105 . 
     An article of manufacture may be used to store program code providing at least some of the functionality of the embodiments described above. Additionally, an article of manufacture may be used to store program code created using at least some of the functionality of the embodiments described above. An article of manufacture that stores program code may be embodied as, but is not limited to, one or more memories (e.g., one or more flash memories, random access memories—static, dynamic, or other), optical disks, CD-ROMs, DVD-ROMs, EPROMs, EEPROMs, magnetic or optical cards or other type of non-transitory machine-readable media suitable for storing electronic instructions. Additionally, embodiments of the invention may be implemented in, but not limited to, hardware or firmware utilizing an FPGA, ASIC, a processor, a computer, or a computer system including a network. Modules and components of hardware or software implementations can be divided or combined without significantly altering embodiments of the invention. 
     This specification refers throughout to computational and network environments that include virtual machines (VMs). However, virtual machines are merely one example of data compute nodes (DCNs) or data compute end nodes, also referred to as addressable nodes. DCNs may include non-virtualized physical hosts, virtual machines, containers that run on top of a host operating system without the need for a hypervisor or separate operating system, and hypervisor kernel network interface modules. 
     VMs, in some embodiments, operate with their own guest operating systems on a host using resources of the host virtualized by virtualization software (e.g., a hypervisor, virtual machine monitor, etc.). The tenant (i.e., the owner of the VM) can choose which applications to operate on top of the guest operating system. Some containers, on the other hand, are constructs that run on top of a host operating system without the need for a hypervisor or separate guest operating system. In some embodiments, the host operating system uses distinct name spaces to isolate the containers from each other and therefore provides operating-system level segregation of the different groups of applications that operate within different containers. This segregation is akin to the VM segregation that is offered in hypervisor-virtualized environments, and thus can be viewed as a form of virtualization that isolates different groups of applications that operate in different containers. Such containers are more lightweight than VMs. 
     It should be recognized that while the specification refers to VMs, the examples given could be any type of DCNs, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, the example networks could include combinations of different types of DCNs in some embodiments. 
     In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the invention(s) are described with reference to details discussed in this document, and the accompanying drawings illustrate the various embodiments. The description above and drawings are illustrative of the invention and are not to be construed as limiting the invention. References in the specification to “one embodiment,” “an embodiment,” “an exemplary embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but not every embodiment may necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, such feature, structure, or characteristic may be implemented in connection with other embodiments whether or not explicitly described. Additionally, as used in this document, the term “exemplary” refers to embodiments that serve as simply an example or illustration. The use of exemplary should not be construed as an indication of preferred examples, operations, and/or that blocks with solid borders are not optional in some embodiments of the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the present invention. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of embodiments of the present inventions. 
     It will be evident that various modifications may be made thereto without departing from the broader spirit and scope of the invention as set forth in the following claims. For example, the methods described in this document may be performed with fewer or more features/blocks or the features/blocks may be performed in differing orders. Additionally, the methods described in this document may be repeated or performed in parallel with one another or in parallel with different instances of the same or similar methods.