Abstract:
At least one intelligent network interface card (INIC) is coupled to a host computer to offload protocol processing for multiple network connections, reducing the protocol processing of the host. Plural network connections can maintain, via plural INIC ports and a port aggregation switch, an aggregate connection with a network node, increasing bandwidth and reliability for that aggregate connection. Mechanisms are provided for managing this aggregate connection, including determining which port to employ for each individual network connection, and migrating control of an individual network connection from a first INIC to a second INIC.

Description:
REFERENCE TO COMPUTER PROGRAM LISTING APPENDIX  
         [0001]    A Computer Program Listing Appendix is included herewith as a part of the present disclosure, including a recordable Compact Disc (CD-R) Disk containing files and computer program code. All the material on the Compact Disc is hereby expressly incorporated by reference into the present application.  
         COPYRIGHT NOTICE  
         [0002]    A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the reproduction of the patent document or the patent disclosure in exactly the form it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.  
         BACKGROUND  
         [0003]    The present invention relates to network communications, in particular to network systems for which a network node maintains more than one connection with another network node.  
           [0004]    Port aggregation (also called link aggregation, teaming or trunking) is a method by which two or more network connections are grouped together at a multiport network host to create a single logical connection. One purpose of this grouping is to be able to increase bandwidth for that single logical connection without having to increase the bandwidth of any of the physical network channels. For example, full-duplex Ethernet or Fast-Ethernet connections can be grouped in this fashion to avoid or delay upgrading a network infrastructure to Gigabit Ethernet or asynchronous transfer mode (ATM).  
           [0005]    Typically associated with port aggregation is a port fail-over method that ensures that the logical connection is maintained in the event that an individual network link or network interface card (NIC) fails. Such a port fail-over method can also provide increased reliability for that single logical network connection, in comparison with the reliability of a single physical network connection.  
           [0006]    To provide an increased number of connections for a network host, the host may be connected to plural networks with plural NICs. A port aggregated logical connection may in this case involve plural ports of the plural NICs. The use of plural NICs may, however, strain a host central processing unit (CPU) due to the additional network protocol processing required for the additional NICs.  
           [0007]    Even without additional NICs, protocol processing may absorb a large fraction of host CPU cycles. This is because conventionally, data such as a file that is transferred over a network is divided into multiple packets, with each packet having layers of protocol headers that are processed one layer at a time by the CPU of the receiving host computer. Although the speed of CPUs has greatly increased over many years, host protocol processing of network messages such as file transfers can consume most of the available processing power of the fastest commercially available CPU.  
         SUMMARY  
         [0008]    In accordance with the present invention, at least one intelligent network interface card (INIC) is coupled to a host computer to offload protocol processing for multiple network connections, reducing the protocol processing of the host. Plural network connections can maintain, via plural INIC ports and a port aggregation switch, an aggregate connection with a network node, increasing bandwidth and reliability for that aggregate connection. Mechanisms are provided for managing this aggregate connection, including determining which port to employ for each individual network connection, and migrating control of an individual network connection from a first INIC to a second INIC. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWING  
       [0009]    [0009]FIG. 1 is a block diagram of a host computer having plural INICs connected to a network by a port aggregation switch, the host including a port aggregation program that manages the logical connections of the INICs. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0010]    [0010]FIG. 1 is a block diagram of a host computer  20  having a CPU  24 , a memory  21 , storage  23 , a first INIC  22  and a second INIC  25 . Note that, although storage  23  is shown separately from memory  21 , both may simply be separate categories of the same memory. First INIC  22  is coupled to network channels  32  and  33  by network ports  52  and  53 , and second INIC  25  is coupled to network channels  34  and  35  by network ports  54  and  55 . Network ports  52 - 55  each include an encoding/decoding mechanism and a physical interface that is coupled to a respective network channel  32 - 35 . Although FIG. 1 illustrates an embodiment with two INICs each having two ports, more or less INICs each having more or less ports are possible. Network channels  32 - 35 , which may each comprise conductive wires, optical fibers, or wireless transmission media, are coupled to a port aggregation switch  37 .  
         [0011]    The port aggregation switch  37  may be coupled to a number of other network channels  40 - 43 , which may also comprise conductive wires, optical fibers, or wireless transmission media. Although only a few network channels  40 - 43  are shown in this illustration, switch  37  may include tens or hundreds of physical connections. Clients  44 - 47  are coupled to switch  37  via network channels  40 - 43 , respectively. Although represented as a few network channels  40 - 43  directly connected to clients  44 - 47  to facilitate illustration, the network channels  40 - 43  may each include multiple packet-switched interconnections between switch  37  and clients  44 - 47 . Likewise, clients  4447  may represent any network nodes including peer level hosts that may have multiple physical network interfaces. Similarly, the host computer  20  may represent any computing or communication device that has a CPU and a memory and is able to be coupled to a network as a node.  
         [0012]    Network connections, such as Transmission Control Protocol (TCP) connections, may be initiated between the host  20  and any of clients  44 - 47 . The network connections typically define the network addresses and the relevant network ports of both the host  20  and client  44 ,  45 ,  46  or  47 , but do not necessarily define the network path connecting the host and client for those ports, and so may be thought of as logical connections. The port aggregation switch  37  can combine plural network connections, each of which is set up to communicate over a different one of the network ports  52 - 55  and channels  32 - 35 , into a single aggregate interface that communicates with client  44 ,  45 ,  46  or  47  via respective network channel  40 ,  41 ,  42  or  43 .  
         [0013]    The INICs  22  and  25  are connected to the host  20  by a conventional bus  58 , which may be a host bus or an input/output (I/O) bus such as a peripheral component interconnect (PCI) bus. Alternatively, INICs  22  and  25  may be connected to the host  20  by plural I/O buses. For the situation in which bus  58  is an I/O bus, internal INIC memory buses  56  and  57  and host memory bus  59  may be coupled to I/O bus  58  with conventional interface mechanisms. The INICs  22  and  25  have protocol processing mechanisms  26 - 29  that process data link, network and transport layer headers of each packet received by that INIC. INICs  22  and  25  also have respective memories  70  and  72  and respective microprocessors  75  and  77 .  
         [0014]    The host memory  21  contains a conventional protocol processing stack  60  that can be run by CPU  24  to process various communication protocols (e.g., IP, TCP, UDP, IPX, SPX, HTTP, etc.), an ATCP protocol processing stack  62  and an INIC device driver  64 . The ATCP protocol stack  62  is used to offload selected network connections to the INICs  22  and  25  for fast-path processing of messages corresponding to those selected connections, while the conventional stack  60  is available for slow-path processing of other messages. The INIC device driver  64  diverts fast-path packets received from the INICs  22  and  25  to the ATCP stack  62  for processing, such as connection setup. The ATCP stack  62  also intercepts outgoing fast-path messages from being processed by the conventional TCP/IP stack  60 . Source code for an embodiment of the ATCP stack  62  that works with Windows NT is contained in the CD-R Disc provided with this specification, in a folder entitled “nt-parallel-stack.” 
         [0015]    Alternatively, ATCP functions such as creating and handing out fast-path connections to INICs  22  and  25  may be included in an integrated protocol stack that also includes instructions for conventional protocol processing, as described in U.S. patent application Ser. No. 09/514,425, filed Feb. 28, 2000 and incorporated by reference herein. Source code for an integrated Free BSD stack is contained in the CD-R Disk Disc provided with this specification, in a folder entitled “freebsd-integrated-stack.” In another embodiment, fast-path connection setup and tear down may be handled by INICs  22  and  25 , as described in U.S. patent application Ser. No. 09/675,484 and U.S. patent application Ser. No. ______, both filed Sep. 29, 2000, and incorporated by reference herein.  
         [0016]    INIC  22  chooses whether to send a packet received from a network channel  32 - 35  to the host memory  21  for slow-path processing of the headers by the CPU  24  running protocol stack  60  or  62 , or to send the packet data directly to a destination in storage  23 . The fast-path may be selected for the vast majority of data traffic having plural packets per message that are sequential and error-free. The fast-path avoids the time consuming protocol processing of each packet by the CPU  24 , such as repeated copying of the data and repeated trips across the host memory bus  59 . Slow-path processing allows any packets that are not conveniently transferred by the fast-path of the INIC  22  to be processed conventionally by the host  20 .  
         [0017]    In order to provide fast-path capability at the host  20 , a logical connection is first set up with a remote node such as client  44 . This connection initialization may include handshake, authentication and/or other procedures. A communication control block (CCB) is created by the ATCP stack  62  during connection initialization procedures for connection-based messages, such as typified by TCP/IP or SPX/IPX protocols. The CCB includes connection information, such as source and destination addresses and ports. For TCP connections a CCB comprises source and destination media access control (MAC) addresses, source and destination Internet Protocol (IP) addresses, source and destination TCP ports and TCP variables such as timers and receive and transmit windows for sliding window protocols. After a connection has been set up, the CCB is passed by INIC device driver  64  from the host  20  to the INIC memory  70  by writing to a command register in that memory  70 , where it may be stored along with other CCBs in a CCB cache. The INIC  22  also creates a hash table corresponding to the cached CCBs for accelerated matching of the CCBs with packet summaries.  
         [0018]    When a message, such as a file write, that corresponds to the CCB is received by the INIC  22 , a header portion of an initial packet of the message is sent to the host  20  to be processed by the CPU  30  and protocol stack  38 . This header portion sent to the host contains a session layer header for the message, which is known to begin at a certain offset of the packet, and optionally contains some data from the packet. The processing of the session layer header by ATCP stack  62  identifies the data as belonging to the file and indicates the size of the message, which are used by a host  20  file system to reserve a destination for the data in storage  23 . If any data was included in the header portion that was sent to the host, it is then stored in the destination.  
         [0019]    A list of buffer addresses for the destination in storage  23  is sent to the INIC  22  and stored in or along with the CCB. The CCB also maintains state information regarding the message, such as the length of the message and the number and order of packets that have been processed, providing protocol and status information regarding each of the protocol layers, including which user is involved and storage space for per-transfer information.  
         [0020]    Once the CCB indicates the destination, fast-path processing of received packets corresponding to the CCB is available. A packet received subsequently at port  52  is first processed by mechanism  26  to generate the packet summary, a hash of the packet summary is compared with the hash table, and if necessary with the CCBs cached in memory  70 , to determine whether the packet belongs to a message for which a fast-path connection has been set up. Upon matching the packet summary with the CCB, assuming no exception conditions exist, the data of the packet, without network or transport layer headers, is sent by direct memory access (DMA) units to the destination in storage  23  denoted by the CCB, which may for example be a file cache for an application.  
         [0021]    Likewise, fast-path messages to be transmitted from the host  20  to the client  44  are diverted from an application interface to the ATCP protocol processing stack  62 , which sends the message data to the INIC  22  or  25  that is holding the CCB for that message. That INIC references the CCB to prepend TCP and IP headers to data packets and sends the packets on the corresponding network channel. The ATCP stack  62  remains available for slow-path processing of any fast-path type packet or message that has exception conditions. A more detailed discussion of the above-described accelerated communication mechanism, which speeds protocol processing and reduces work for the host CPU  24 , can be found in U.S. Patent Application Serial No. 60/061,809, U.S. patent application Ser. No. 09/067,544, U.S. Patent Application Serial No. 60/098,296, U.S. patent application Ser. No. 09/141,713, U.S. patent application Ser. No. 09/384,792, U.S. patent application Ser. No. 09/439,603, U.S. patent application Ser. No. 09/464,283, U.S. patent application Ser. No. 09/692,561, U.S. patent application Ser. No. 09/748,936 and the U.S. patent application Ser. No. ______ filed Feb. 20, 2001, by Express Mail No. EF055069864US, inventors Laurence B. Boucher et al., entitled “Obtaining a Destination Address so that a Network Interface Device can Write Network Data Without Headers Directly into Host Memory,” all of which are incorporated by reference herein.  
         [0022]    In accordance with a port aggregation protocol, the port aggregation switch  37  controls which network ports  52 - 55  are associated with a network such as channel  40 . That is, port aggregation switch  37  may move a connection from one to another of ports  52 - 55 . Since the fast-path conditions described above involve offloading control and processing of a connection to INIC  22  or  25  in association with ports  52  and  53  or  54  and  55 , respectively, the fast-path and port aggregation protocol need to be synchronized.  
         [0023]    A port aggregation and fail-over scheme that may be used by switch  37  is referenced in IEEE standard 802.3ad, which is incorporated by reference herein. A similar type of port aggregation and fail-over scheme is called “Fast Etherchannel,” promoted by Cisco Systems®. Fast Etherchannel combines plural network ports into a single logical interface. In the Fast Etherchannel implementation, each of the ports in the logical group shares the same MAC address. Because of this, each of the ports is connected to a single Fast Etherchannel switch (such as the Cisco Catalyst™ series switch). If a link such as one of channels  32 - 35 , ports  52 - 55  or INICs  22 , 25  fails in a fast Etherchannel group, host  20  and switch  37  each independently identify the link failure and switch to another link. Alternatively, port aggregation switch  37  may attempt to balance the traffic on the network ports  5255  that are associated with network channels  40 - 43 .  
         [0024]    A port aggregation driver  66  is disposed between the INIC device driver  64  and the protocol processing stacks  60  and  62  to handle the port aggregation requirements imposed by the switch  37 . For example, if the switch  37  migrates a fast-path connection from port  52  on INIC  22  to port  54  on INIC  25 , the port aggregation driver  66  can recognize the migration and transfer the corresponding CCB from first INIC  22  to second INIC  25 . Source code for the port aggregation driver  66  is contained in the CD-R Disk provided with this specification, in a folder entitled “pag” located in the folder entitled “nt-parallel-stack.” 
         [0025]    The port aggregation driver  66  is transparent to upper protocol layers such as TCP/IP stack  60 , ATCP stack  62 , or integrated Free BSD stack. That is, the upper protocol layers are not aware that they are communicating across a logical group of network interfaces. This is illustrated in FIG. 1 with the single arrow leading between TCP/IP stack  60  and port aggregation driver  66 , compared with four arrows leading between port aggregation driver  66  and INIC device driver  64 . Likewise, a single arrow leading between ATCP stack  62  and port aggregation driver  66  illustrates the communication between plural ports  52 - 55  of plural INICs  22 ,  25  and the single ATCP stack  62 . The INIC device driver  64  can control INICs  22  and  25  with signals flowing from port aggregation driver  66 . For the situation in which port aggregation is not being used, the port aggregation driver  66  is not active.  
         [0026]    As mentioned above, port aggregation and fail-over switching mechanisms are provided across multiple INICs notwithstanding individual INIC control and processing of each fast-path connection. Thus a fast-path message transfer can be interrupted by port aggregation switch  37  deciding to move a fast-path connection to another INIC. Communicating a message using a fast-path connection may involve a large block of data, such as a Server Message Block (SMB) write or read, that is divided into multiple 64 kilobyte (KB) messages, which are further divided into multiple 1.4 KB packets for network transfer according to IP. For example, host  20  may wish to issue a write to client  44 , for a connection corresponding to a CCB held on INIC  22 . INIC  22  will split the data into TCP packets and transmit the packets according to the TCP sliding window protocol. In order to do this, INIC  22  also processes the acknowledgments returned by the client. Since the CCB for the connection resides on INIC  22 , it is helpful for all acknowledgments for that CCB to be sent to INIC  22 .  
         [0027]    As noted above, however, the port aggregation switch  37  may be configured to decide which of the network channels  32 - 35  and ports  52 - 55  is to receive a particular packet. It is difficult in this port aggregation environment for host  20  or INICs  22  and  25  to predict a priori on which port a packet for a given logical connection will arrive. Instead, the port aggregation driver  66  monitors the network traffic received by INICs  22  and  25  to keep track of which logical connections, identified by the MAC address of client  44 , are associated with which of the ports  52 - 55 .  
         [0028]    With information regarding the port  52 ,  53 ,  54  or  55  that is associated with a logical connection for client  44 , the port aggregation driver  66  can mimic the port aggregation switch  37  by handing out a CCB to the port  52 ,  53 ,  54  or  55  associated with the destination MAC address. This information may be gleaned, for example, from a connection initialization handshake or from initial replies to a write or read request. Until the port aggregation driver  66  has identified which port is associated with a logical connection, fast-path processing of a message corresponding to that connection is averted. Averting fast-path processing may be accomplished simply by the port aggregation driver  66  identifying CCB handout attempts of the ATCP stack  62  and failing them until the driver  66  has identified which of the ports  52 ,  53 ,  54  or  55  is associated with the connection. The ATCP driver  62  may be configured to delay subsequent handout attempts to avoid thrashing. Once a port  52 ,  53 ,  54  or  55  is associated with a connection then the CCB handout is allowed to succeed, providing fast-path processing of messages, such as the SMB write, by INIC  22  or  25 .  
         [0029]    After this fast-path processing has begun, however, port aggregation switch  37  may change the port selection for load balancing purposes, so long as the switch can guarantee that packets are not sent out of order. This is one mechanism by which INIC  22 , for example, can receive a packet for a fast-path connection that is being handled by the other INIC  25 .  
         [0030]    In this case the INIC  22  that receives the packet cannot process the packet according to the fast-path connection, and instead sends the packet to the INIC device driver  64 , which is configured to divert fast-path type message packets to the ATCP stack  62  for processing. The ATCP stack  62  maintains a list of the CCBs that have been offloaded to INICs  22  and  25 , and recognizes that this slow-path packet corresponds to a CCB that is in a fast-path state. Upon receiving this exception condition, the ATCP stack  62  will command the INIC  25  to flush the fast-path CCB back to the ATCP stack  62 . After the packet has been processed by the ATCP stack  62  and the state of the CCB updated to reflect that processing, the CCB can then be handed out to the INIC  22 , which is known by port aggregation driver  66  to be associated with the connection.  
         [0031]    When the port aggregation driver  66  receives a slow-path send request, it extracts the destination MAC address from the packet to determine which INIC and corresponding port  52 ,  53 ,  54  or  55  should be used to send the packet. For send requests corresponding to a CCB held by an INIC  22  or  25 , the port aggregation driver  66  may not receive this information. Instead, a connection handle is created to identify a particular fast-path connection. The connection handle is in one embodiment a 4-byte value made up the following four 1-byte values:  
         [0032]    1. Connection identifier—This identifies the CCB on the INIC  22  or  25 . In one embodiment, up to 256 CCBs can be held per INIC.  
         [0033]    2. INIC number—This identifies the INIC (e.g.,  22  or  25 ) associated with the fast-path connection.  
         [0034]    3. Port number—This identifies the port (e.g.,  52 ,  53 ,  54  or  55 ) associated with the connection by its number on a given INIC.  
         [0035]    4. Generation number—A number used for indicating INIC failure, discussed below.  
         [0036]    The connection handle is set by the INIC device driver  64  during CCB handout and passed back up to the ATCP stack  62  as an opaque handle. The ATCP stack  62  uses this handle for all subsequent fast-path requests for that logical connection.  
         [0037]    During connection handout, the ATCP stack  62  provides the destination MAC address as part of the handout. The port aggregation driver  66  intercepts the destination MAC address as the request is being passed down from the ATCP stack  62  to the INIC device driver  64 . Similarly, the port aggregation driver  66  intercepts the connection handle as the completion is passed back up from the INIC device driver  64  to the ATCP stack  62 . The port aggregation driver  66  uses information from the MAC address and connection handle to identify which fast-path requests belong to which port and INIC.  
         [0038]    Other issues that are solved in accordance with the present invention include the handling of a link failure for a fast-path connection. There are two ways in which a link failure can occur. One way is for the host  20  to receive a link status signal indicating that the link has failed. Another way is for the INIC handling the link (or links) to crash. Both of these fail-over scenarios are discussed separately below.  
         [0039]    For connections that are operating in slow-path mode, handling a link failure is simple. Link failure is identified by the INIC  22  or  25 , which notifies the INIC device driver  64  via an interrupt status register. The INIC device driver  64  in response issues a media disconnect status indication to the protocol drivers above it, including the port aggregation driver  66 . Upon receiving the media disconnection indication, the port aggregation driver  66  notes the affected port  52 ,  53 ,  54  or  55  and refrains from sending subsequent slow-path packets out that port. Until a new port is associated with a particular connection (as described above) the port aggregation driver selects an outgoing port based on the lower bits of the destination MAC address.  
         [0040]    Ownership of connections by INIC  22  or  25  complicates handling a link failure for fast-path mode connections. If a link failure results in the connection being associated with a link on the other INIC  22  or  25 , the CCB is flushed back to the host  20  and then handed out to the other INIC. The port aggregation driver  66  may include instructions to flush the fast-path CCB back to the host  20  when a link fails.  
         [0041]    Alternatively, instructions on the port aggregation switch  37  and ATCP stack  62  may manage the link failure without intervention by the port aggregation driver  66 . In this case, link failure may be handled by different mechanisms. First, the port aggregation switch  37  may discover the link failure and switch the connection to a new port  52 ,  53 ,  54  or  55 . If the new port is on a different INIC  22  or  25 , then the ATCP stack  62  will receive slow-path packets for a fast-path connection, in which case it will flush the CCB from the INIC associated with the link failure. Second, a TCP retransmission timer on the INIC may be triggered, also causing the CCB to be flushed to the host from the INIC associated with the link failure.  
         [0042]    Certain operating system controls, however, may interfere with the above mechanisms. For example, Windows NT or  2000  network driver interface specification (NDIS), upon receiving an indication from INIC device driver  64  that a link has failed, may prohibit protocol stacks such as ATCP  62  from sending commands such as a flush command to the INIC device driver. For this situation, the INIC device driver  64  instead may be configured to issue a flush command to the appropriate INIC  22  or  25  when alerted of a link failure by that INIC.  
         [0043]    Failure of one of the INICs  22  or  25  is more difficult to manage. The difficulty is in recovering the CCBs that have been offloaded to the failed INIC. If the INIC  22  or  25  is no longer functional, then the INIC cannot flush the CCBs back to the host. It may be possible to read the CCBs out of SRAM on the INIC, but if the state of the INIC is suspect, then the state of the CCBs is also suspect. Instead, a safer procedure is to close all CCBs on the failed INIC.  
         [0044]    Some upper layer protocols, such as Netbios, reopen connections automatically. As such, a host  20  with SMB mapped file systems should not experience a loss of connectivity. Other sessions, such as File Transfer Protocol (FTP), may have to be reestablished by the host  20 . One challenge is for the ATCP stack  62  to determine which connections need to be terminated and which do not. Although it may be possible to explicitly tell the ATCP stack  62  which connections need to be flushed, this may involve the INIC device driver  64  issuing some sort of custom status indication to the ATCP stack  62 , which may be undesirable.  
         [0045]    Instead, the INIC device driver  64  maintains a generation number, as mentioned above, for each INIC  22  and  25 . This generation number gets incremented every time the INIC gets reset. The generation number gets passed up to the ATCP driver as part the previously mentioned connection handle during CCB handout. On every subsequent fast-path request, the ATCP stack  62  passes this opaque handle back down to the INIC device driver  64 . If the INIC device driver  64  gets a request with a stale generation number, as the result of an INIC reset, the INIC device driver  64  will fail the fast path request. When the ATCP stack  62  discovers that the fast-path request failed it will know that it must abort the TCP connection.  
         [0046]    Note that it is possible that the ATCP stack  62  already has an outstanding command on INIC  22  or  25  at the time that INIC fails. Without further information, the ATCP stack  62  could end up waiting indefinitely for the command to complete. For this reason, the ATCP stack  62  implements a fast-path command timeout. When the timeout expires, the ATCP stack  62  will attempt to flush the connection. If the flush fails (due to the generation number) or times out, then it will abort the connection.  
         [0047]    Although we have described in detail various embodiments of the present invention, other embodiments and modifications will be apparent to those of skill in the art in light of this text and accompanying drawings. Therefore, the present invention is to be limited only by the following claims, which include all such embodiments, modifications and equivalents.