State migration in multiple NIC RDMA enabled devices

A method is provided for transferring control between a first network interface and at least a second network interface in a same multiple network interface device after the first network interface transmits an identifier generated by the first network interface. The method includes receiving a message from the second network interface to a program component, the message indicating the reception of the identifier from a second device. Next, the method provides for querying the first network interface to supply the program component with a list of identifiers generated by the first network interface and associated memory locations in the multiple network interface device memory. If the identifier received by the second device is present in the list, the method provides for transmitting a memory location associated with the identifier to the second network interface.

FIELD OF THE INVENTION

This invention relates generally to networking and, more particularly, relates to more efficiently use of CPU resources in a networked devices.

BACKGROUND OF THE INVENTION

Networked computing has become almost ubiquitous. Many computers in use today have multiple network interface controllers, or NICs. The speed of the data transfers allowed by these NICs has grown faster than the CPU processing power and memory system bandwidth available to process this data. Consequently, more of this processing has been moved into the NIC card themselves. While this solution works to a point, additional standards activities have been underway to develop protocols which further assist in offloading the burden of network processing from the CPU and memory bus to the NIC.

One such protocol is remote direct memory access or RDMA. RDMA is a protocol which allows the NIC card to place a data packet in a predetermined memory location in the computer systems main memory. In the standard network protocol stack, the RDMA software resides just above the transport control protocol (TCP) software. This allows a data packet to be placed directly in system memory with minimal intervention from the CPU.

The RDMA protocol is used to make a section of main system memory on a first machine directly available to a remote second machine. The protocol associates the memory in the first machine with a handle referred to as a STag. To offload as much processing as possible from the CPU in the first machine, a NIC in the first machine generates the STag generated by a consumer comprising, for example, an application program. The STag is then sent to the second machine, which can perform a write by sending the STag back with associated data. Upon receiving this data and associated STag, the NIC in the first machine will read the STag and use a DMA transfer to move the data into the memory associated with the STag.

Data traveling over the internet can take several different routes from one machine to another. The path through the Internet will change when loading on servers changes or when servers fail all together. This can cause difficulty for a machine with multiple NICs when performing an RDMA transfer. As the route the data takes through the Internet changes, it is possible that the path chosen from the first machine to the second machine will change in a manner that causes the path between these two machines to change from NIC 1 to NIC 2 in machine 1. Recall that the NIC 1 generates the STag. Therefore, NIC 2 will have no knowledge of a STag generated by NIC 1. If the route from machine 1 to machine 2 uses NIC 1 when an STag in generated, and then the route changes to one which uses NIC 2 before machine 2 sends data to machine 1, machine 2 will return data with an STag that is unknown to NIC 2.

There is a need for a method to handle STag's generated by one NIC and received by another NIC in the same machine.

BRIEF SUMMARY OF THE INVENTION

Embodiments are directed to methods that overcome the problem of a STag arriving at a network interface that did not generate the STag. The method relies on network interfaces on a given computer having unique STags. This can be assured by the operating system. Because STags on a given computer are unique, a network interface receiving an STag generated by another network interface on the same computer is enabled to detect that the STag was generated by a different network interface. When such a STag is detected, the network interface receiving the STag passes this STag to a higher level of software, which can be an RDMA program component which resides in the OS kernel. The RDMA program component can identify which NETWORK INTERFACE generated the STag and query the associated network interface for all STags generated by this network interface and the associated addresses of the allocated memory. The address is then passed to the network interface that received the unknown STag. With the memory address, the network interface can then complete the data transfer.

More specifically, an embodiment is directed to a method for transferring control between a first network interface and at least a second network interface in a multiple network interface device after the first network interface transmits an identifier generated for use by the first network interface to a second device. The identifier can be associated with a memory location in the multiple network interface device, and the identifier and an associated data field are capable of being received by the second network interface. The method further includes receiving a message from the second network interface to a program component, the message indicating the reception of the identifier from the second device. Next, the method provides for querying the first network interface to supply the program component with a list of identifiers generated by the first network interface and associated memory locations in the multiple network interface device memory. If the identifier received by the second device is present in the list, the method provides for transmitting a memory location associated with the identifier to the second network interface. Thus, the second network interface becomes capable of transmitting the associated data field to the memory location associated with the identifier.

Another embodiment is directed to a method for transferring control between a first network interface and at least a second network interface in a host computer including the first network interface and the second network interface. The method includes receiving an identifier from a remote computer, the identifier generated by the first network interface and associated with a memory location in the host computer. Next, the method provides for sending a message to a program component indicating the reception of the identifier, the program component configured to query the first network interface for a list of identifiers generated by the first network interface and associated memory locations in the host computer. If the list of identifiers includes the identifier from the remote computer, the method provides for receiving a memory location associated with the identifier. If the list of identifiers does not include the identifier from the remote computer, the method provides for invalidating the identifier from the remote computer.

Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments, which proceeds with reference to the accompanying figures.

DETAILED DESCRIPTION OF THE INVENTION

Referring toFIG. 1, in its most basic configuration, the computing device100includes at least a processing unit102and a memory104. Depending on the exact configuration and type of computing device, the memory104may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. The basic configuration is illustrated inFIG. 1by a dashed lines106. Additionally, the device100may also have additional features/functionality. For example, the device100may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tapes. Such additional storage is illustrated inFIG. 1by a removable storage108and a non-removable storage110. Computer storage media includes volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. The memory104, the removable storage108and the non-removable storage110are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the device100. Any such computer storage media may be part of the device100.

Device100may also have one or more input devices114such as keyboard, mouse, pen, voice input device, touch-input device, etc. One or more output devices such as a display, speakers, printer, etc. may also be included. All these devices are well known in the art and need not be discussed at greater length here.

Remote direct memory access is a protocol for more efficiently transporting data in a data network. This protocol was adopted by an industry consortium and approved by the consortium in October of 2002. The RDMA protocol can be used on top of the ubiquitous TCP/IP protocol. RDMA allows the transfer of data directly into a systems memory without the need for host processor intervention, thus greatly reducing the burden of data transfer on the host processor.

Remote direct memory access (RDMA) is a protocol approved by an industry consortium. It has been submitted to the Internet Engineering Task Force (IETF) for approval as an IETF standard. The goal of RDMA is to reduce the loading on the CPU required to handle network traffic. Reducing CPU loading has become necessary because the data rate of networks has grown faster than the available processing power of host CPU's and available memory bandwidth. For example, 10 Gb/s ethernet is becoming an industry standard while as recently as 10 years ago 10 Mb/s ethernet was common.

RDMA allows one machine to place data directly into the memory of another machine with minimal intervention from the CPU and minimal demands on the available memory bandwidth. RDMA has been designed to work transparently over standard TCP/IP networks in a specific version referred to as RDMA over TCP/IP. This is accomplished by embedding RDMA specific control fields to the data to be transmitted to form an RDMA packet and then embedding the RDMA packet in a TCP/IP datagram. Thus, to the network the TCP/IP packet appears the same as any other TCP/IP packet and is handled in an identical manner as a TCP/IP packet without the RDMA header would be handled.

Advances in network interface controllers (NICs) have allowed RDMA to become useful. The first such advance was the TCP/IP offload engine (TOE). The TOE technique moves much of the TCP/IP processing onto the NIC, relieving the host CPU of much of this burden. However, TOE does not by itself does not always allow zero copy of the incoming data meaning that often even if the TOE NIC uses DMA to transfer the incoming data to memory the data will still need to be copied by the network layer software. The need for this additional copy is highly dependent on the application programmers interface (API) required by the NIC and the software interfacing to the NIC. RDMA is designed to reduce or eliminate the need for this copy and therefore reduce the need for CPU intervention.

RDMA accomplishes the zero copy operation by allocating a memory location for a particular data transfer before initiating the transfer. The memory location is associated with an identifier referred to as a STag. The STag is sent as part of the RDMA header and is sent to a remote machine. If RDMA over TCP/IP is being used then this header is embedded in a TCP/IP packet. The remote machine will then return the data and the same STag embedded in another TCP/IP packet. The RDMA and TOE-enabled NIC card on the originating machine will process the TCP/IP portion of the packet and realize that there is an embedded STag in the data. It can then look up this STag, find the associated memory location, and place the data directly into this memory location using DMA. This differs from non-RDMA transfer in that a TCP/IP packet alone without RDMA would not have the associated memory location and would in general have to be passed into software to determine the final destination of the data. Once this destination was determined the software might have to use CPU resources to copy the data to a buffer appropriate for the destination application. With RDMA, this destination can be determined before the data arrives and hence no copy is required.

Security is a significant issue which was addressed by the RDMA consortium. Using RDMA, a remote computer can place commands directly into the memory of the host computer. One method used to combat this security issue is to allow STags to be invalidated after a transfer is completed. To this end, the RDMA header includes a bit to invalidate the STag. If this bit is set in a received packet, the STag is removed from a list of active STag's and any further RDMA packets received with this STag are discarded. The STag itself is a 32-bit number and if STags are quickly invalidated the chances of a malicious attempt to guess an active STag are small. While the RDMA standard does not require that STags be invalidated after a given number of transfers, for security reasons most applications using RDMA will invalidate a given STag after every data transfer is completed.

Flowchart200of a simplified RDMA read cycle is shown inFIG. 2. One of skill in the art with the benefit of this disclosure will appreciate that the simplified RDMA read cycle shown is exemplary for purposes of the below discussion. In flowchart200, the data transferred over the network is assumed to use the TCP/IP protocol. It will be appreciated by one of ordinary skill in the art that other transmission protocols are possible. First is the generation of an identifier referred to as an STag in block201. The STag is then associated with data storage, typically main system memory in block203. Blocks201and203combined are referred to as binding. Next, the STag and a read request from an RDMA packet which have been encapsulated in a TCP/IP datagram are sent to a remote machine as a request to acquire data from the remote machine in block205. The STag is concatenated with the desired data field in the remote machine in block207. The concatenation of the STag and the data field is then returned to the originating machine. Once received, the STag is removed from the TCP/IP datagram and the RDMA packet and the associated memory location is ascertained in block209. Once the memory location is known, the data is transferred to this location and becomes available to the application which initiated the data transfer in block211.

FIG. 3is a block diagram of a networked computer system300capable of implementing a typical RDMA read transfer. Computer301is the host computer and will be used to exemplify the RDMA transfer. Computer301consists of memory system302, typically including but not limited to random access memory (RAM), application program303, operating system305, and NIC309. These components are not intended to represent all the components in a typical computer. Instead, these components are those necessary to carry out an RDMA transfer. Further, memory system302contains an allocated block of memory which is memory allocation311. For purposes of this description, this data storage will be referred to as memory and is usually RAM in present computer systems. However, the scope of the invention is not limited to RAM and may in general be any data storage device including but not limited to FLASH, hard disk, RAM, or any other storage device which can perform a read and a write. Computer system301is also referred to as a computer for purposes of this description, however this can in general be any device which can support a network interface including but not limited to PDAs, computers, cell phones, and set top boxes.

Referring now toFIG. 3andFIG. 4in combination, the flow diagram ofFIG. 4illustrates an RDMA read operation. The transfer is initiated by an application303making a request of OS305to transfer data in block402. All operating system commands in this case can occur in kernel mode. The OS305, which may include network layer307, determines that the request requires a network access. If NIC309is capable of RDMA, OS305can make this transfer utilizing RDMA to offload some of the processing burden from the host CPU. The RDMA transfer starts by OS305requesting a STag from NIC309in block404. NIC309will return an identifier referred to as STag401to network OS305in block406. OS305will then allocate memory for the transfer creating memory allocation311and send the address of the allocated memory311to NIC309in block408. NIC309then associates memory allocation311with STag401. This association allows NIC309to place any data arriving with STag401in memory allocation311. NIC309then creates RDMA packet403in block412. RDMA packet403consists of STag401and a command to read data. NIC309next encapsulates RDMA packet403in a TCP/IP datagram405in block414. In block416, TCP/IP datagram405is transmitted onto the Internet313which routes the TCP/IP datagram405to remote computer315in block416. Remote computer315extracts RDMA packet403from TCP/IP datagram405in block418. This extraction can be done in a NIC or in software. In block420, STag401is extracted from RDMA packet403and in combination with the requested data407is used to form RDMA packet411. In block422TCP/IP datagram409encapsulating RDMA packet411TCP/IP datagram409is then sent onto Internet313which routes TCP/IP datagram409to computer301in block424. NIC309then receives TCP/IP datagram409in block426. NIC309then extracts RDMA packet411from TCP/IP datagram409in block428. In block430NIC309extracts STag401from RDMA packet411and checks the invalidation bit in the RDMA header. If this bit is set, the STag is invalidated. STag401is then used to retrieve the associated memory allocation311and the requested data407is sent to this memory allocation using DMA in block432.

Note that the entire transfer of data required very little CPU intervention. OS305needs only make a request that the transfer take place to NIC309, perform memory allocation311, and report the location of this memory to NIC309. NIC309handles all data transfers. Since the memory is allocated per transfer, the memory can be allocated in a manner that allows the application to directly access the data. The direct access by the application prevents the data from having to be moved after the NIC places the data in memory. This prevents the CPU from having to move a large amount of data which would consume a great deal of CPU resources in a high speed network.

FIG. 5shows a block diagram500of an RDMA read transfer between two networked computers in accordance with the present invention. Block diagram500is identical to block diagram300except that computer501has a second NIC510. Like numbers are used to indicate like components. System memory502contains an allocated block of memory which is memory allocation511. Operating system505may include network layer507.

Referring toFIG. 5in combination withFIGS. 6aand6b, a flow diagram illustrates an RDMA read operation in accordance with an embodiment. The flow diagram makes reference to identifying numbers used inFIG. 5for purposes of clarity. The transfer follows the same procedure as that shown inFIG. 4until TCP/IP datagram is returned from the remote computer. In the transfer represented inFIGS. 6aand6bhowever, the route used by the Internet has changed such that TCP/IP datagram609takes a route which terminates in NIC510instead of NIC509which initiated the transfer. Because NIC510did not initiate the transfer, NIC510has no knowledge of STag601and associated memory location511. This knowledge does, however, exist in NIC509.

FIG. 6aillustrates that an application503that may be implemented in remote computer501may request transfer of data from operating system OS505, in step602. In step604, OS505may request STag from NIC509, in response to which NIC509returns STtag601to OS505, in step605. OS505then may allocate memory location511and notify NIC509about it, in step608, upon which NIC509may associate memory location511with STag601, in step610. In step612, NIC509forms RDMA packet603with the read request. In step614, NIC509forms TCP/IP datagram611encapsulating RDMA packet603. The TCP/IP datagram611may then be sent by NIC509to remote computer515over the Internet513, in step616. Upon receiving the TCP/IP datagram611, remote computer515may extract RDMA packet603from TCP/IP datagram611, in step618. In step620, remote computer515forms an RDMA packet606with STag601and requested data607. In step622, remote computer515may form TCP/IP datagram609encapsulating RDMA packet606. In step624, remote computer515sends TCP/IP datagram609to computer501over the Internet513. As was discussed above, the route used by the Internet may change so that, in step626, second NIC (e.g., NIC510) may receive TCP/IP datagram from second device.

When NIC510receives TCP/IP datagram609in block626ofFIG. 6b, NIC510will perform all necessary TCP/IP processing and then extracts RDMA packet606in block628. Next, STag601is removed from RDMA packet606in block630. In block632, NIC510will search a list of all valid STags created in NIC510but will fail to find STag601. There are two possibilities at this point, either STag601is invalid or STag601was created by another NIC on the same computer501. NIC510assumes that the later is true and, in block634, reports STag601to OS505as an unrecognized STag. OS505attempts to ascertain if STag601is valid. To accomplish validation, in block636OS505queries NIC509for all valid STags and associated addresses generated by NIC509. In block638NIC509returns the requested list. OS505then searches this list for STag601, in block640. In block642, OS505makes a decision as to whether STag601is found. If not, block650is implemented in which STag601is reported as invalid to NIC510. NIC510will then discard the packet because there is no valid memory location to transfer the packet. If a valid memory location were available, transferring a packet with an invalid STag into memory would present a security risk. Note that while in this example this branch is clearly not possible as we stated that NIC509generated STag601. It is included however to illustrate the complete algorithm which must correctly handle invalid STags. If block642determines that STag601is in the list of STags generated by NIC509, then, in block644, OS505finds memory location511associated with STag601. In block646associated memory location511is reported to NIC510. The knowledge of associated memory allocation511allows NIC510to transfer requested data607to memory allocation511completing the transfer in block648.

During the switch from one NIC to another, the CPU must actively participate in the processing of some of the RDMA protocol stack. If this situation continued indefinitely it would clearly be undesirable. Further, there may be many outstanding STags waiting to be processed on NIC509that will be affected by a route change. In the worst case, say, for example when a cable was unplugged from NIC509, then all remaining STags in NIC509would be processed by NIC510. The RDMA standard does not set any time limits on the lifespan of STags, therefore it is possible that the CPU would need to be involved with the use of these STags indefinitely. In practice, however, STag's are usually invalidated after each transfer and only rarely used for a large number of transfers. Therefore, in most cases, the software involvement in the transfer of STags is for a very limited duration. The invalidation of a STag can be accomplished by setting the invalidate flag in the RDMA header, though it can also occur by an explicit command from a software application. For security reasons, most applications using RDMA will invalidate a STag after one transfer even if the invalidate bit was not set.

In one embodiment of the invention, remote computer515also includes an application that allocates an STag, associates it with memory on remote computer515, and sends it to the application503on local computer501. Application503then performs an RDMA Write. More specifically, an RDMA Write packet is created which contains the remote STag as well as the data. The data and remote STag are encapsulated in a TCP/IP datagram and sent to the remote computer515. An RDMA NIC on remote computer515receives this datagram, and via the STag, knows which location in memory to place the data. RDMA NIC places this data in that memory via DMA. An IP route change can affect this STag just as the case for an RDMA Read explained above and remote computer515can handle the route change in the same manner as that explained above for an RDMA Read.

Likewise, for simplicity, all of the above descriptions dealt with a single STag. In some embodiments, the RDMA Read operation involves two STags. Referring back toFIG. 2, even before step201, an application on the remote computer315(inFIG. 3) allocates an STag (call it “remote Stag”) and associates it with memory on computer315. Then, the application on remote computer315sends this remote STag to the application on the local computer301. This allows the local application303to perform an RDMA Read (blocks201-211inFIG. 2). The RDMA Read Request packet (block205) actually carries two STags: the local STag as well as the remote STag. IP route changes can affect remote STags in the same manner as local STags. In that case, handling of the remote STag by the remote computer315, is exactly the same as handling of the local STag by the local computer301.