Abstract:
A hash table in the network device driver maintains data on the traffic characteristics for each network interface (“NIC”) within a computing device. If one of the NICs in the computing device becomes unreliable, the cost function in the hash engine allows the software driver to initiate network traffic redistribution among the remaining reliable NICs in the computing device. Using this hash engine, the software driver is able to intelligently redirect each of the network connections on an unreliable NIC to a reliable NIC within the computing device, in a way that optimizes the distribution of network traffic across the remaining reliable NICs. Alternatively, if a connection is moved from an old NIC to a new NIC, the software driver can detect the moved connection and offload the moved connection to a hardware offload engine on the new NIC. With this approach, issues such as network interface overloading and computing device performance degradation may be more easily avoided when failing over network connections, thereby improving overall system performance relative to prior art techniques.

Description:
BACKGROUND OF THE INVENTION 
   1. Field of the Invention 
   Embodiments of the present invention relate generally to network communications and more specifically to a system and method for intelligently failing over network connections in a load-balanced networking environment. 
   2. Description of the Related Art 
   Performance and reliability are key requirements for modern computer networks. When a network interface (“NIC”) on a computing device is no longer reliable, the networking performance of that NIC may drop substantially, possibly to zero. In such situations, contemporary solutions redirect network connections from the unreliable NIC to other, reliable NICs on the same computing device, a technique referred to as “failover.” Although failover may mitigate the immediate problem of the existence of an unreliable NIC, the additional network traffic on the remaining reliable NICs from the failed-over connections may cause one or more of the remaining reliable NICs to become overloaded, thereby reducing the performance of those interfaces. Additionally, the additional network traffic on the remaining reliable NICs may substantially increase the network processing performed by the computing device, thereby reducing the performance of the computing device. 
   As the foregoing illustrates, what is needed in the art is a technique for failing over network connections to one or more reliable NICs that reduces the likelihood of NIC overloading or other phenomena that can impair overall system performance. 
   SUMMARY OF THE INVENTION 
   One embodiment of the present invention sets forth a method for offloading a network connection from a host software module to a first hardware offload engine included in a failover network interface card (NIC) in a switch-independent network environment. The method includes the steps of requesting the identity of the failover NIC, determining the identity of the failover NIC from the output of a hash algorithm, setting an entry in a first delegated connection table included in the first hardware offload engine, where the entry includes connection state for the network connection, and setting a hardware identifier in a connection table included in the host software module that corresponds to the network connection, where the set hardware identifier indicates that the network connection is being run through the first hardware offload engine. 
   One advantage of the disclosed method is that issues such as network interface overloading may be more easily avoided when failing over network connections, thereby improving overall system performance relative to prior art techniques 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments. 
       FIGS. 1A and 1B  illustrate a computing device in which one or more aspects of the present invention can be implemented; 
       FIG. 2  illustrates a flowchart of method steps for unoffloading network connections from a failed NIC to software; 
       FIGS. 3A and 3B  illustrate a flowchart of method steps for offloading network connections from software to a failover NIC in a switch-independent network environment; and 
       FIG. 4  illustrates a flowchart of method steps for offloading a network connection to a new NIC when the network connection is moved to the new NIC by external means. 
   

   DETAILED DESCRIPTION 
   Intelligent failover of network connections from a failed NIC to one or more operational NICs may be accomplished by using a hash engine that tracks network statistics, including the number of connections and amount of transmitted and received traffic through each NIC in a switch-independent (i.e., a network environment in which a switch coupling the NICs of computing device  100  to an external network is not operating in an 802.3ad mode) network environment. Once a failed or unreliable NIC has been detected, the hash engine is queried for an alternative NIC that may be underutilized or otherwise able to accommodate the connections from the failed NIC. Such a query allows one or more additional network connections to be handled by the alternative NIC without exceeding the capacity of that NIC, thereby avoiding a failover operation that may substantially reduce the network performance of the computing device associated with the NICs. Additional network performance benefits may be realized by unoffloading the connections from the hardware offload engine within a failed NIC and subsequently offloading those connections to the hardware offload engine(s) within one or more failover NICs. 
   In another embodiment of the invention, the selection of a failover NIC is performed external to the computing device, potentially by a switch in a switch-dependent (i.e., a network environment in which a switch coupling the NICs of computing device  100  to an external network is operating in an 802.3ad mode) network environment or by an external computing device in a switch-independent network environment. Since the process for selecting a failover NIC in this embodiment differs from that for selecting a NIC in switch-independent environments, this embodiment is considered separately. In addition, since the benefits from offloading a network connection from a software TCP/IP (Transmission Control Protocol &amp; Internet Protocol) stack to a hardware offload engine are well known to those skilled in the art, these issues will not be discussed herein. 
     FIGS. 1A and 1B  illustrate a computing device  100  in which one or more aspects of the present invention can be implemented. As shown, the computing device  100  includes a main memory  102 , a memory controller  104 , a microprocessor  106 , an I/O controller  108 , and NICs  110 ,  118 . NIC  110  includes a multicast list  116  and a hardware offload engine (“HOE”)  112 , which further includes a delegated connection table (“DCT”)  114 . NIC  118  includes a multicast list  124  and an HOE  120 , which further includes a DCT  122 . HOEs  112 , 120  include logic configured for processing network frames associated with network connections between the computing device  100  and one or more remote network computing devices (not shown) that have been selectively offloaded to NICs  110 ,  118 . By processing network frames with logic in the NICs  110 , 118  (sometimes referred to as “handling connections in hardware”) rather than performing those processing functions in a host software TCP/IP stack (sometimes referred to as “handling connections in software”) as is conventionally done, communications between the NICs  110 ,  118  and the microprocessor  106  as well as computations performed by the microprocessor  106  may be substantially reduced. 
   DCTs  114 , 122  are configured to include a plurality of delegated connection table entries, where each entry corresponds to a network connection whose frames are processed by the HOE that includes that particular DCT. In one embodiment of the invention, each DCT entry includes the following state information for each delegated connection: destination (i.e., remote computing device) internet protocol (“IP”) address, destination port number, source (computing device  100 ) port number, and certain TCP/IP connection state necessary to run the delegated connection in hardware. 
   The memory controller  104  is coupled to the main memory  102  and to the microprocessor  106 , and the I/O controller  108  is coupled to the microprocessor  106  and the NICs  110 , 118 . In one embodiment of the invention, the microprocessor  106  transmits commands or data to the NICs  110 , 118  by writing commands or data into the I/O controller  108 . Once such commands or data are written into the I/O controller  108 , the I/O controller  108  optionally translates the commands or data into a format that the target NIC may understand and communicates the commands or data to the target NIC. Similarly, NICs  110 , 118  transmit commands or data to the microprocessor  106  by writing commands or data into the I/O controller  108 , and the I/O controller  108  optionally translates the commands or data into a format that the microprocessor  106  may understand and communicates the commands or data to the microprocessor  106 . The aforementioned couplings may be implemented as memory busses or I/O busses, such as PCI™ busses, or any combination thereof, or may otherwise be implemented in any other technical feasible manner. 
   As shown in more detail in  FIG. 1B , the main memory  102  includes an operating system  126  and a software driver  128 . The software driver  128  includes a Load Balancing and Failover (“LBFO”) module  130  and a TCP/IP stack  134 . LBFO module  130  tracks networking status for each NIC (e.g., by monitoring the link status of each NIC, the number of send and receive errors on each NIC and/or whether each NIC is sending and receiving keep-alive packets) and communicates with the TCP/IP stack  134  when network traffic is being moved from a failed NIC to software or from software to a failover NIC. The LBFO module  130  includes a hash engine  132 , which intelligently determines how network connections should be distributed across the different functional NICs in the computing device  100 , based on the aforementioned networking statistics, as described in the related U.S. patent application titled, “Intelligent Failback In a Load-Balanced Networking Environment,” filed on May 18, 2007 and having Ser. No. 11/750,914. This related patent application is hereby incorporated herein by reference. The TCP/IP stack  134  includes a connection table  136 , which includes a plurality of connection table entries, where each entry corresponds to a network connection between the computing device  100  and a remote computing device. In one embodiment, each connection table entry includes the following state information for each connection: the destination IP address, destination port number, source IP address, source port number and a hardware identifier for the connection as well as enough TCP/IP state to handle dropped packets and the like. Importantly, the hardware identifier for a particular connection indicates which HOE, if any, is processing that connection in hardware. In one embodiment, the connection state for each connection being handled in hardware may be routinely copied from the delegated connection table within the HOE charged with processing the connection to the connection table  136 . For example, the state information may be copied using a repeating timer. In this fashion, an updated copy of the connection state for each delegated connection may be maintained in the connection table  136  for use by the TCP/IP stack  134 . 
   The computing device  100  may be a desktop computer, server, laptop computer, palm-sized computer, personal digital assistant, tablet computer, game console, cellular telephone, or any other type of similar device that processes information. 
     FIG. 2  illustrates a flowchart of method steps  200  for unoffloading network connections from a failed NIC to software, according to one embodiment of the invention. Although the method is described in reference to the computing device  100 , persons skilled in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present invention. 
   As shown, the method for unoffloading network connections begins at step  202 , where the LBFO module  130  monitors the status of each NIC  110 ,  118  for an indication that one of the NICs has either failed or become unreliable. In one embodiment, a NIC is determined to have failed or become unreliable when any of three conditions are present. First, the LBFO module  130  may determine that there is a link disconnection for the NIC, suggesting that a network cable has become disconnected, based on the link indication for that NIC. Second, the LBFO module  130  may determine that a substantial number or percentage of packets transmitted by the NIC were lost, based on a comparison between the number of packets sent by the NIC and the number of packets acknowledged by the remote computing devices to which the packets were transmitted. Third, the LBFO module  130  may determine that keep-alive packets transmitted between the NICs  110 ,  118  in the computing device  100  are being lost. As described in the related U.S. patent application titled, “Technique for Identifying a Failed Network Interface Card within a Team of Network Interface Cards,” filed on Dec. 15, 2005 and having Ser. No. 11/303,285, the failed NIC can be identified based on how each NIC is transmitting and/or receiving the keep-alive packets. This related patent application is hereby incorporated herein by reference. By monitoring each NIC for these types of indications, the LBFO module  130  is capable of reliably detecting a failed or unreliable NIC. 
   In step  204 , the LBFO module  130  determines whether a NIC has failed or become unreliable based on the indications being monitored in step  202 . If the LBFO module  130  determines that no failure has occurred, then the method returns to step  202 , where the LBFO module  130  continues monitoring the NICs  110 ,  118  for a failure. 
   If, however, a NIC is found to have failed or become unreliable, the method proceeds to steps  206 - 214 , where the connections being run through that NIC are unoffloaded to software. For purposes of discussion only, it is assumed that NIC  110  has failed. In step  206 , the LBFO module  130  signals the TCP/IP stack  134  that NIC  110  has failed or has become unreliable. In step  208 , the LBFO module signals the hash engine  132  that NIC  110  has failed or has become unreliable to avoid having any new connections assigned to the failed NIC  110 . Upon receiving this signal, the hash engine  132  adjusts its internal load balancing algorithm so that new connections are not assigned to the failed NIC  110 . 
   In step  210 , for each connection being run through the failed NIC  110 , the HOE for the failed NIC (e.g., HOE  112 ) invalidates the entry in the DCT (e.g., DCT  114 ) corresponding to its connection. In one embodiment, the DCT entry is invalidated by the TCP/IP stack  134  sending a command to the HOE  112  to invalidate the DCT entry, the HOE  112  receiving the command and invalidating the DCT entry, and the HOE  112  acknowledging the completion of this operation to the TCP/IP stack  134 . In step  212 , for each connection being unoffloaded from the failed NIC  110 , if the connection state in the DCT  114  for that connection does not match the connection state in the connection table  136 , then the TCP/IP stack  134  copies the connection state for the connection from the DCT  114  to the connection table  136 . In step  214 , for each connection being unoffloaded from the failed NIC  110 , the TCP/IP stack  134  invalidates the hardware identifier for that connection in the connection table  136 . Invalidating the hardware identifier indicates that the connection is being run through the TCP/IP stack  134 . Importantly, once the connection state for a connection has been invalidated in the DCT  114  and the hardware identifier for the connection has been invalidated in the connection table  136 , any packets transmitted or received on that connection are handled thereafter in software through the TCP/IP stack  134 , rather than in hardware, through the HOE  112 . Once all connections on the failed NIC  110  have been unoffloaded from hardware to software in this fashion, the method terminates in step  216 . 
     FIGS. 3A and 3B  illustrate a flowchart of method steps for offloading network connections from software to a failover NIC in a switch-independent network environment, according to one embodiment of the invention. Although the method is described in reference to the computing device  100 , persons skilled in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present invention. Importantly, in a switch-independent network environment, the LBFO module  130  identifies a failover NIC for each connection unoffloaded to software, as described above in  FIG. 2 . 
   As shown, the method for offloading a single network connection from software to a failover NIC is described in steps  302 - 312 . These steps are repeated for each connection unoffloaded to software in method  200  such that all unoffloaded connections are eventually offloaded from software to one or more failover NICs. In step  302 , the TCP/IP stack  134  requests the LBFO module  130  to identify a NIC to which the current unoffloaded connection should be offloaded. Again, as discussed in step  208 , the hash engine  132  tracks the status of each NIC in the computing device  100  and adjusts its connection assignment algorithm when a NIC fails or becomes unreliable to avoid having any new or existing connections offloaded to that failed or unreliable NIC. In step  304 , the LBFO module  130  queries the hash engine  132  to identify a suitable NIC to which the current connection should be offloaded. For purposes of discussion only, it is assumed that the hash engine  132  identifies the failover NIC  118  as the NIC to which the current connection (which had previously been unoffloaded from the failed NIC  110 ) should be offloaded. 
   In step  306 , the software driver  128  adds the MAC address of the failed NIC  110  to the multicast list  124  of the failover NIC  118 , which allows the failover NIC  118  to receive packets addressed to the MAC address of the failed NIC  110 . In step  308 , the TCP/IP stack  134  directs the failover NIC  118  to send a learning packet to the switch. The learning packet may be any technically feasible packet type that includes the MAC address of the failed NIC  110 . As is well-known, sending such a packet from the failover NIC  118  causes the switch to reconfigure itself to route subsequent packets destined for the MAC address of the failed NIC  110  to the failover NIC  118 . Thus, all network traffic related to the current connection being offloaded to the failover NIC  118  is thereafter received by the failover NIC  118 . As described in greater detail in the related patent application having, a consequence of this approach is that all connections from a particular failed NIC are offloaded to the same failover NIC. 
   In step  310 , HOE  120  sets the connection state in the DCT  122  corresponding to the current connection being offloaded. Again, setting the connection state in a DCT entry includes setting its destination IP address, destination port number, the source port number and certain TCP/IP state for the connection being offloaded. In one embodiment, the DCT entry is set by the TCP/IP stack  134  sending a command to the HOE  120  to set the connection state, the HOE  120  receiving the command and setting the connection state, and the HOE  120  acknowledging the completion of this operation to the TCP/IP stack  134 . In step  312 , for the connection being offloaded, the TCP/IP stack  134  sets the hardware identifier within the connection table  136  to a device handle that represents the HOE  120  of the failover NIC  118 . The hardware identifier indicates that the connection is being run through the HOE  120 . Importantly, once the hardware identifier for a connection is set to a device handle for a particular HOE, any packets transmitted or received on that connection are handled thereafter in hardware through that HOE. 
   In step  314 , the TCP/IP stack  134  determines whether all unoffloaded connections for the failed NIC  110  have been offloaded to a failover NIC. If all unoffloaded connections have not been offloaded to a failover NIC, the method returns to steps  302 - 312 , where another unoffloaded connection is offloaded to a failover NIC. If, however, all unoffloaded connections have been offloaded to a failover NIC, the method terminates in step  316 . 
     FIG. 4  illustrates a flowchart of method steps for offloading a network connection to a new NIC when the network connection is moved to the new NIC by external means, according to one embodiment of the invention. Although the method is described in reference to the computing device  100 , persons skilled in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present invention. 
   As previously described, failover can occur when a connection is moved from an old NIC to a new NIC (i.e., the failover NIC) in at least two situations. One situation involves a switch-dependent network environment, in which a switch determines that a NIC has failed or has become unreliable and therefore moves some or all connections associated with the failed or unreliable NIC to one or more failover NICs. Another situation involves a switch-independent network environment, where the packets received for a connection by the computing device  100  are addressed to and received by a NIC (i.e., the new/failover NIC) other than the old NIC. Here, some computing device in the network other than computing device  100  has taken the initiative to transmit packets associated with the connection to the new NIC. In both situations, the LBFO module  130  recognizes that a new NIC has been selected for a connection when the LBFO module  130  detects that packets for the connection are being received on a NIC other than the old NIC. As described in greater detail below, the LBFO module  130  monitors the NICs for incoming packets, determines whether incoming packets indicate a moved connection, unoffloads the moved connection from the old NIC, and offloads the moved connection on the new NIC. For purposes of discussion only, it is assumed that NIC  110  is the old NIC. 
   As shown, the method for offloading a network connection begins at step  402 , where the LBFO module  130  monitors the NICs for network traffic, as described above. In step  404 , for a given connection, the LBFO module  130  detects that packets for the connection have been received on a new NIC rather than on the old NIC  110 . Again, this may occur in at least two situations. The first is where, in a switch-dependent network environment, the switch detects a failed or unreliable NIC and moves the connection to a failover NIC. The second is where, in a switch-independent network environment, a computing device starts sending packets for a connection to a NIC other than the old NIC, on which the packets for the connection were initially received. For purposes of discussion only, it is assumed that the new NIC identified in step  404  is NIC  118 . In step  406 , the software driver  128  unoffloads the connection from the old NIC  110  to software, as previously described above with respect to  FIG. 2 . 
   In step  408 , the LBFO module  130  communicates the identity of the new NIC  118  to the hash engine  132 , which causes the hash engine  132  to update its internal data structures for the additional connection on the new NIC  118 . In step  410 , HOE  120  sets the connection state in the DCT  122  corresponding to the connection being offloaded to NIC  118 . Again, setting the connection state in a DCT entry includes setting its destination IP address, destination port number, the source port number and certain TCP/IP state for the connection being offloaded. In one embodiment, the DCT entry is set by the TCP/IP stack  134  sending a command to the HOE  120  to set the connection state, the HOE  120  receiving the command and setting the connection state, and the HOE  120  acknowledging the completion of this operation to the TCP/IP stack  134 . In step  412 , for the connection being offloaded, the TCP/IP stack  134  sets the hardware identifier within the connection table  136  to a device handle that represents the HOE  120  of the new NIC  118 , before returning to step  402 . Again, the hardware identifier indicates that the network connection is being run through the HOE  120 . As previously described herein, once the hardware identifier for a connection is set to a device handle for a particular HOE, any packets transmitted or received on that connection are handled thereafter in hardware through that HOE. 
   One should note that after a connection associated with an old NIC  110  has been unoffloaded to software pursuant to method  200 , but before the connection is offloaded to a new NIC pursuant to method  400 , the computing device  100  may transmit packets for this connection from software on a NIC temporarily selected by the hash engine  132 . In general, the selection of a temporary NIC is performed by the LBFO engine  130  querying the hash engine  132  for a NIC whose current transmit performance is at a level such that transmitting the packets associated with an additional connection would not substantially degrade the transmit performance for the existing connections on that NIC. Once the LBFO engine  130  detects that a packet associated with the unoffloaded connection being run out of software has been received on a new NIC, as described above, the transmission of packets for this connection on the temporary NIC is discontinued and the connection is offloaded to the new NIC, for transmit and receive packets, pursuant to method  400 . 
   One advantage of the disclosed method is that using a hash engine to intelligently distribute network traffic among the different functional NICs in the computing device  100 , when failing over connections in switch-independent network environment, may substantially improve network performance relative to prior art solutions. Another advantage of the disclosed method is that offloading a network connection to a failover NIC when the network connection is moved to the failover NIC may substantially improve computing device performance relative to prior art solutions. 
   While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof. The scope of the present invention is determined by the claims that follow.