Method and apparatus for performing connection management with multiple stacks

The disclosed embodiments relate to a communication device for use in a node of a system having a plurality of nodes. Each of the plurality of nodes may include network interface controllers (“NICs”) and each of the NICs may have a public identifier and a private identifier associated therewith. A first protocol stack may operate according to a first protocol that is associated with the public identifier and a second protocol stack may operate according to a second protocol that is associated with the private identifier. A storage device may associate the public identifier of one or more of the NICs with the first protocol stack and the private identifier of one or more of the NICs with the second protocol stack. Received messages that incorporate the public identifier may be directed to the first protocol stack and messages that incorporate the private identifier may be directed to the second protocol stack.

BACKGROUND OF THE RELATED ART

In the field of computer systems, it may be desirable for information to be transferred from a system memory associated with one computer system to a system memory associated with another computer system. Communication between computer systems may involve exchanging and processing messages through a proprietary protocol stack at each of the computer systems. However, these proprietary networks may not be compatible with other networks or systems that employ different communication protocols.

If multiple protocols are used to facilitate communication within networks, packets may be mishandled. For instance, a packet that was formatted under a first protocol may be incorrectly interpreted as being formatted under a second protocol. In such a case, the information contained within the mishandled packets may be misdirected or lost.

DESCRIPTION OF SPECIFIC EMBODIMENTS

The Remote Direct Memory Access (“RDMA”) Consortium, which includes the assignee of the present invention, is developing specifications to improve the ability of computer systems to remotely access the memory of other computer systems. One such specification under development is the RDMA Consortium Protocols Verb specification, which is hereby incorporated by reference. The verbs defined by this specification may correspond to operations or actions that may form an interface for data transfers between memories in computer systems, including the formation and management of queue pairs, memory windows, protection domains and the like.

RDMA may refer to the ability of one computer to directly place information in the memory space of another computer, while minimizing demands on the central processing unit (“CPU”) and memory bus. In an RDMA system, an RDMA layer may interoperate over any physical layer in a Local Area Network (“LAN”), Server Area Network (“SAN”), Metropolitan Area Network (“MAN”), or Wide Area Network (“WAN”).

Referring now toFIG. 1, a block diagram illustrating a computer network in accordance with embodiments of the present invention is illustrated. The computer network is indicated by the reference numeral100and may comprise a first processor node102and a second processor node110, which may be connected to a plurality of I/O devices126,130,134, and138via a switch network118. Each of the I/O devices126,130,134and138may utilize a Remote Direct Memory Access-enabled Network Interface Card (“RNIC”) to communicate with the other systems. InFIG. 1, the RNICs associated with the I/O devices126,130,134and138are identified by the reference numerals124,128,132and136, respectively. The I/O devices126,130,134, and138may access the memory space of other RDMA-enabled devices via their respective RNICs and the switch network118.

The topology of the network100is for purposes of illustration only. Those of ordinary skill in the art will appreciate that the topology of the network100may take on a variety of forms based on a wide range of design considerations. Additionally, NICs that operate according to other protocols, such as InfiniBand, may be employed in networks that employ such protocols for data transfer.

The first processor node102may include a CPU104, a memory106, and an RNIC108. Although only one CPU104is illustrated in the processor node102, those of ordinary skill in the art will appreciate that multiple CPUs may be included therein. The CPU104may be connected to the memory106and the RNIC108over an internal bus or connection. The memory106may be utilized to store information for use by the CPU104, the RNIC108, or other systems or devices. The memory106may include various types of memory such as Static Random Access Memory (“SRAM”) or Dynamic Random Access Memory (“DRAM”).

The second processor node110may include a CPU112, a memory114, and an RNIC116. Although only one CPU112is illustrated in the processor node110, those of ordinary skill in the art will appreciate that multiple CPUs may be included therein. The CPU112may be connected to the memory114and the RNIC116over an internal bus or connection. The memory114may be utilized to store information for use by the CPU112, the RNIC116or other systems or devices. The memory114may utilize various types of memory such as SRAM or DRAM.

The switch network118may include any combination of hubs, switches, routers and the like. InFIG. 1, the switch network118comprises switches120A-120C. The switch120A connects to the switch120B, the RNIC108of the first processor node102, the RNIC124of the I/O device126and the RNIC128of the I/O device130. In addition to its connection to the switch120A, the switch120B connects to the switch120C and the RNIC132of the I/O device134. In addition to its connection to the switch120B, the switch120C connects to the RNIC116of the second processor node110and the RNIC136of the I/O device138.

Each of the processor nodes102and110and the I/O devices126,130,134, and138may be given equal priority and the same access to the memory106or114. In addition, the memories may be accessible by remote devices such as the I/O devices126,130,134and138via the switch network118. The first processor node102, the second processor node110and the I/O devices126,130,134and138may exchange information using one or more communication protocols. The exchange of information using multiple protocols is explained with reference toFIG. 2.

FIG. 2is a block diagram illustrating the adaptation of a consumer with multiple protocols in accordance with embodiments of the present invention. The block diagram of a consumer with multiple protocols is indicated by the reference numeral150. The RNICs108,116,124,128,132and136(FIG. 1) may be adapted to exchange information using multiple protocols.

A consumer151, which may comprise a process or application, may interact with two different protocol layer stacks. The first protocol layer stack may include an upper layer protocol (“ULP”)152, which may interact with a kernel bypass protocol160. Examples of protocols that may be used for the kernel bypass protocol160include the WinSock Direct (“WSD”) protocol, the Sockets Direct Protocol (“SDP”) or the like. The kernel bypass protocol160may interact with an RDMA protocol154. The RDMA protocol154may interact with a direct data placement protocol (“DDP”)156. The kernel bypass protocol160, the upper layer protocol152, the RDMA protocol154and the DDP156may be employed to bypass the kernel of the operating system (“OS”) of the device that hosts the RNIC.

The bypass protocol160may allow unmodified socket applications to enhance performance of the system by utilizing features of the RDMA protocols, such as protocol offload, OS bypass, true zero copy of data. The kernel bypass protocol160may employ kernel bypass protocol stacks to optimize network performance. The use of the kernel bypass stacks may allow increased bandwidth efficiency, lowered messaging latency and conserving processor time for use by applications. Thus, the kernel bypass stacks may improve the data transfers for systems within the network.

The DDP protocol156may translate messages from the RDMA protocol154for transmission across a network, such as switch network118(FIG. 1). Also, the DDP protocol156may receive messages from other nodes and translate those messages for transmission using the RDMA protocol154. The term iWARP may be used to refer to the suite of protocols comprising the RDMA protocol154, the DDP protocol156and a marker with protocol data unit alignment (“MPA”) protocol (not shown) which may be layered with the bypass protocol160.

For other messages, the consumer151may interact with a second protocol stack, such as a communication protocol158, which may include the transmission control protocol/internet protocol (“TCP/IP”) or the like. In addition to the IP protocol, routing information may be provided by a routing protocol such as AppleTalk, DEC Net or the like. The communication protocol158may comprise other protocols, such as the User Datagram Protocol (“UDP”) or the like. Another communication protocol may be used to provide message framing within the TCP byte stream by using a fixed interval marker mechanism, such as the MPA protocol. The MPA protocol may include a length, may add a backward marker at a fixed interval to segments of upper level data, and/or may add cyclical redundancy check (“CRC”) information. The operation of the communication protocol158and the kernel bypass protocol160is further explained with respect toFIG. 3.

FIG. 3is a block diagram of a multiple protocol stack configuration in accordance with embodiments of the present invention. The block diagram is generally indicated by the reference numeral300. A first node302and a second node304, which may correspond to any of the processor nodes102or110, or the I/O devices126,130,134or138shown inFIG. 1, may be connected via a network306. The network306may correspond to the switch network118ofFIG. 1. The nodes302and304may exchange packets or messages across the network306using various protocols, such as the communication protocol158or the bypass protocol160(FIG. 2). Each of the nodes302and304may include various components to manage the exchange of messages through various protocols, such as a multiple stack configuration with each of the stacks corresponding to a unique media access control (“MAC”) address.

The first node302comprises an RNIC308and the second node304comprises an RNIC328. The RNIC308comprises a physical port component310and the RNIC328comprises a physical port component330. The physical port components328and330may receive and transmit data packets across the network306. The RNICs308and328may communicate using the iWARP suite of protocols. These protocols may employ packets that contain source addresses as well as destination addresses, which may include multiple MAC addresses for each of the respective nodes302or304. The physical port component310or330may be logically divided to support one or more of the upper level components, such as one of the multiple stacks or other components in the RNIC308or328.

The network components312and332may also manage other functions, such as an address resolution protocol (“ARP”), a dynamic host configuration protocol (“DHCP”), and an Internet group management protocol (“IGMP”). ARP may be a used to dynamically resolve a high level IP address to a low-level hardware address across a physical network. DHCP may provide a framework for passing configuration information to hosts on a network, which may add the capability of automatic allocation of reusable network addresses and additional configuration options. IGMP may allow a node302or304to report its multicast group membership to adjacent routers or network equipment to allow the node302or304to send information to other nodes302or304that have identified themselves as interested in receiving the information. Through the RNICs308and328, a first stack314(for the RNIC308) may be connected to a first stack334(for the RNIC328), while a second stack316(for the RNIC308) may be connected to a second stack336(for the RNIC328).

The first stacks314and334may be protocol stacks used to manage communication according to the communication protocol158(FIG. 2). The first stacks314and334may provide standard (non-proprietary) methods for protocol addressing, sending and receiving datagrams, writing and reading on streams, and/or detecting disconnects for interfacing with an application-programming interface (“API”). Also, the first stacks314and334may provide connection-oriented service or port for-a specific application to use in communicating with other nodes.

The second stacks316and336may be protocol stacks used to manage communication according to the kernel bypass protocol160(FIG. 2). The second stacks316and336may comprise a WSD stack or SDP stack that may use RDMA features to bypass the kernel and to reduce the load on a processor within the node302or304. A graphical user interface (“GUI”) may be implemented to interact with an API and WSD stack. The GUI may translate information to associate an IP address to the stack. The second stack316or336may provide proprietary methods for protocol addressing, sending and receiving datagrams or messages, writing and reading on streams, and/or detecting disconnects for interfacing with an API. Also, the second stacks316and336may provide connection-oriented service or port for a specific application to use in communicating with other nodes302or304.

The first node may comprise a memory320and the second node304may comprise a memory340. The memories320and340may include various types of memory, including static read only memory (“SRAM”) or dynamic read only memory (“DRAM”). For purposes of illustration, the memory320may correspond to the memory106(FIG. 1) and the memory340may correspond to the memory114(FIG. 1). The memories320and340may store, among other things, IP or MAC addresses associated with the communication protocol158and the kernel bypass protocol160(FIG. 2). Each of the RNICs may have a separate MAC and IP address assigned thereto for each of the communication protocol158(FIG. 2) and the kernel bypass protocol160(FIG. 2). The IP address associated with the communication protocol158(FIG. 2) may be public and the IP address associated with the kernel bypass protocol160(FIG. 2) may be private. When a data packet arrives at the RNIC308or328, the IP address in the packet directs the packet to be processed by the first stacks314or334if the IP address of the packet corresponds to the communication protocol158(FIG. 2). When a data packet arrives at the RNIC308or328, the IP address in the packet directs the packet to be processed by the second stacks316or336if the IP address of the packet corresponds to the kernel bypass protocol160(FIG. 2).

The memory320of the node302may store a first lookup table322and a second lookup table324. The first lookup table322and the second lookup table324may be accessible by the second protocol stack316, which is associated with the kernel bypass protocol160(FIG. 2). The memory340of the node304may store a first lookup table342and a second lookup table344. The first lookup table342and the second lookup table344may be accessible by the second protocol stack336, which is associated with the kernel bypass protocol160(FIG. 2).

The first lookup tables322and342may include a local address list that may comprise public IP address and a corresponding private IP address associated with local devices or nodes. The second lookup tables324and344may include a remote address list that may comprise public IP addresses and corresponding private IP addresses for remote devices. The second lookup tables324and344may grow as IP addresses for newly discovered remote devices are added. In some embodiments of the invention, the first and second lookup tables322,324,342, and344for each node may be unified. The IP addresses stored in the first lookup tables322and342and the second lookup tables324and344may be included in packets that are being sent from their respective nodes using the kernel bypass protocol160(FIG. 2), which is associated with the second protocol stacks316and336. In other words, the first lookup tables322and342and the second lookup tables324and344may associate the IP addresses of the associated RNIC to the MAC address of the RNIC for purposes of communication using the kernel bypass protocol160(FIG. 2).

The MAC address corresponding to the stacks314,316,334and336may be manually or automatically entered. The MAC address for each stack may be created based on information in the associated memory (320or340), a setting associated with the physical port component (310or330), or from information elsewhere within the respective node (302or304). Each MAC address may have an associated routing address, such as an IP address mapped thereto. Along with the MAC addresses, a multicast group address may be defined for each of the stacks314,316,334, or336and may include the various stacks314,316,334, or336. The multicast group address may be may be manually entered for each of the stacks314,316,334, or336or may be automatically determined. The multicast group address may be used in connection with the kernel bypass protocol160(FIG. 2).

Various requests or operations may be used to manage and/or populate the lookup tables322,324,342, and344. For instance, to join a group, a node302or304may transmit a “Join Group” IGMP message to allow the node302to become a member of the multicast group. Also, ARP requests may be directed to the nodes302or304and may be handled through the RNIC308or328to populate the second lookup tables324and344, which contain information about remote devices. Once a node302or304becomes active, it may send out a multicast message to the multicast group address. The message may be a “MAP Request” or “Update MAP Request” message. The message may include flags, such as add, valid, or delete, which are associated with the addresses.

If statically defined addresses are used, the lookup tables322,324,342, and344may be manually configured to include or add other addresses. For instance, the addresses may be assigned through a GUI interface, a registry, or from within the memory320or340. Thus, with either static or dynamic addressing, the lookup table322,324,342and344may be managed to allow the nodes302or304to communicate through the appropriate stacks314,316,334, and336.

Advantageously, by utilizing the lookup tables322,324,342and344, the nodes302and304may be able to manage the packets and direct the packets to the appropriate stack within a node302or304. In addition, the nodes operating with WSD enabled stacks may not be limited to proprietary network and may operate on a heterogeneous network306. Furthermore, the mapping or connection establishment mechanism may enable certain packets to be directed to specific stacks that allow the node302or304to operate in an enhanced manner over existing networks, while not having an adverse effect on the existing networks. Accordingly, a system employing one or more of the disclosed embodiments may exchange information with other systems faster because of the connection establishment mechanism.

FIG. 4is a process flow diagram illustrating the processing of a received packet in accordance with embodiments of the present invention. In the diagram, generally referred to by reference numeral400, a connection establishment mechanism may be implemented and may be utilized in a system, such as a computer system. The process begins at block402. At block404, a message or packet may be received at a node. The message may be a WSD packet, a TCP packet, an ARP message, an IGMP request, a “MAP Update Request” message, a “MAP Request” message or the like. The node may be a computer system or node302or304(FIG. 3) that includes multiple stacks. The stacks may be the first stacks314or334and the second stacks316or336(FIG. 3). One of the stacks may be a communication protocol stack, such as the communication protocol158(FIG. 2), while the other stack may be a kernel bypass protocol, such as the kernel bypass protocol160. Then, as shown in block406, the RNIC of the node may examine the packet to determine the MAC address. Then, the node may relate the MAC address of the packet to an IP address at block408. Once the IP address is identified, the RNIC may access a lookup table to determine if the IP address is associated with the first stack or the second stack at block410. The lookup table may be a lookup table that includes the mappings of local addresses. For instance the lookup table may be the lookup table322or342(FIG. 3).

At block412, the RNIC may determine if the IP address is associated with a first stack IP address or a second stack IP address. If the IP address does not correspond to the second stack, the packet may be further processed by a first stack, such as first stack314or334(FIG. 3), at block414. However, if the IP address does correspond to the second stack IP address, the packet may be further processed by a second stack, which may be the second stack316or332(FIG. 3) at block416. In either block414or416, the packet may be used to perform various functions or may include information for the node. After either block414or416, the process may end, as shown at block418.

FIG. 5is a process flow diagram illustrating the processing of a sent packet in accordance with embodiments of the present invention. In the diagram, generally referred to by reference numeral500, a connection establishment mechanism may be implemented and may be utilized in a system, such as a computer system, to enable the system to communicate with other similarly enabled nodes. The process begins at block502. At block504, a message or packet may be created in an upper layer protocol, such as an application or API. The message may be an operation or information and may involve communication designated for a specific stack, such as stacks314,316,334, or336(FIG. 3), in a multiple stack system. The message may include destination information for a specific node or group of nodes. The destination information may be an IP address, MAC address, or multicast group IP address, and/or MAC address.

At block506, the node may determine the IP address for the destination node for the message. The node may lookup the IP address from a section of memory or lookup table within the systems memory, which may be the second lookup table324or344(FIG. 3). The memory or lookup table may include IP addresses for other nodes and may map the public IP addresses to the private IP addresses. The public IP addresses may correspond to a TCP/IP stack or other communication stack158(FIG. 2), while the private IP addresses may correspond to a WSD stack or other kernel bypass stack160(FIG. 2). In addition, to the IP addresses additional information may be included within the tables, such as MAC addresses or other information.

At block508, the node may determine if the IP address is within memory. If the IP address is within the memory, then the node may determine if the IP address is in the second stack at block510. However, if the IP address is not in memory, then the request may be directed to the first stack for processing at block514. At block510, the node may determine if the IP address is in the second stack. If the second stack has the IP address within a table or memory, such as the second lookup table324or344(FIG. 3), then the node may process the packet at the second stack in block520. However, if the second stack does not have the IP address, then the request may be directed to the first stack for processing at block514.

The message may be may be prepared for transmission at the first stack, which may be in a RNIC, such as RNIC308or328(FIG. 3). Accordingly, the packet may be processed at the RNIC with protocol layers at block516. In preparing the message, the IP address for the intended recipient may be mapped to a MAC address and included with the message before being transmitted at block518.

At the second stack, the node may further process the packet, as shown at block520. The node may determine if the IP address is within a table, such as the second lookup table324or344(FIG. 3). If the IP address is within the table, the packet may be further processed in block522at the RNIC associated with the node, such as RNIC308or328(FIG. 3). In preparing the message, the IP address for the intended recipient may be mapped to a MAC address and included with the message before being transmitted at block518. The packet may be transmitted to another node with an IP address within the table. Accordingly, the process may end, as shown at block524.