Abstract:
The invention relates to insertion and removal of MPA markers and RDMA CRCs in RDMA data streams, after determining the locations for these fields. An embodiment of the invention comprises a host interface, a transmit interface connected to the host interface, and a processor interface connected to both transmit and host interfaces. The host interface operates under the direction of commands received from the processor interface when processing inbound RDMA data. The host interface calculates the location of marker locations and removes the markers. The transmit interface operates under the direction of commands received from the processor interface when processing outbound RDMA data. The transmit interface calculates the positions in the outbound data where markers are to be inserted. The transmit interface then places the markers accordingly.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application is a Continuation of U.S. Non-Provisional Application No. 11/415,181, filed May 2, 2006, incorporated by reference herein in its entirety, and for which priority is claimed under 35 U.S.C. §120. 
     
    
     BACKGROUND OF THE INVENTION 
       [0002]    1. Field of the Invention 
         [0003]    The invention described herein relates to data communications and, in particular, direct memory access (DMA). 
         [0004]    2. Related Art 
         [0005]    Remote DMA (RDMA) is a technology for transferring data from the memory of one computer or server to the memory of another, without involving a CPU or operating system of either machine. Because the data being transferred is not stored in application memory or in operating system buffers, RDMA is said to accomplish the transfer in a “zero-copy” manner. 
         [0006]    RDMA is typically implemented using a suite of three protocols—RDMA Protocol (RDMAP), Direct Data Placement (DDP) and Marker PDU Aligned Framing Protocol (MPA). RDMAP provides interfaces to applications for sending and receiving data. DDP slices outgoing data into segments that fit into TCP&#39;s Maximum Segment Size (MSS), and places incoming data into destination buffers. MPA provides a framing scheme that facilitates DDP operations in identifying DDP segments. 
         [0007]    RDMA is a “shim”, a transport protocol suite on top of TCP. RDMA leverages TCP rather than inventing its own protocols for flow control, routing, data sequencing and so on. In principle, an RDMA message can be too large to fit into one TCP segment. 
         [0008]    MPA is a framing protocol. It adds a marker into the data stream at a stride of every 512 bytes in the TCP sequence space. Markers assist the receiver in locating the DDP/RDMA header. 
         [0009]    Unfortunately, insertion and removal of MPA markers are not friendly operations. Inserting markers into a continuous data stream creates a disruptive shuffle of the data stream. Insertion and removal of a RDMA CRC (cyclic redundancy code) digest is also difficult to handle efficiently. 
         [0010]    Therefore there is a need for a system and apparatus with which MPA markers and CRC digests can be easily inserted and removed during RDMA communications. 
       SUMMARY OF THE INVENTION 
       [0011]    The invention described herein inserts and removes MPA markers and RDMA CRCs in RDMA data streams, after determining the locations for these fields. An embodiment of the invention comprises a host interface, a transmit interface connected to the host interface, and a processor interface connected to both transmit and host interfaces. 
         [0012]    The host interface operates under the direction of commands received from the processor interface when processing inbound RDMA data. The host interface calculates the location of marker locations and removes the markers. CRCs are handled in a like manner. 
         [0013]    The transmit interface operates under the direction of commands received from the processor interface when processing outbound RDMA data. The transmit interface calculates the positions in the outbound data where markers are to be inserted. The transmit interface then places the markers accordingly. CRCs are handled in a like manner. 
         [0014]    Further embodiments, features, and advantages of the present invention, as well as the operation of the various embodiments of the present invention, are described below with reference to the accompanying figures. 
     
    
     
       BRIEF DESCRIPTION OF THE FIGURES  
         [0015]    The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present invention and together with the description further serve to explain the principles of the invention and to enable a person skilled in the pertinent art to make and use the invention. In the drawings, like reference numbers indicate identical or functionally similar elements. Additionally, the left-most digit of the reference number indicates a drawing in which the reference number first appears. 
           [0016]      FIG. 1  illustrates the structure of RDMA communications in the context of the transmission control protocol. 
           [0017]      FIG. 2  illustrates the relationship between an RDMA—capable network interface card and a host, according to an embodiment of the invention. 
           [0018]      FIG. 3  is a block diagram of the invention as it could be implemented in the form of an integrated circuit. 
           [0019]      FIG. 4  illustrates a processor interface, according to an embodiment of the invention. 
           [0020]      FIG. 5  illustrates a host interface, according to an embodiment of the invention. 
           [0021]      FIG. 6  illustrates a transmit interface, according to an embodiment of the invention. 
           [0022]      FIG. 7  illustrates a packet format before and after processing by a data formatter, according to an embodiment of the invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0023]    An embodiment of the present invention is now described with reference to the figures. While specific configurations and arrangements are discussed, it should be understood that this is done for illustrative purposes only. A person skilled in the relevant art will recognize that other configurations and arrangements can be used without departing from the spirit and scope of the invention. It will be apparent to a person skilled in the relevant art that this invention can also be employed in a variety of other systems and applications. 
       Introduction 
       [0024]      FIG. 1  offers a schematic view of data framing with respect to RDMA. When an application issues a command for data transfer, RDMA considers all data  110  identified by the application when forming an RDMA message  120 . 
         [0025]    DDP is responsible for slicing a large RDMA message  120  into smaller segments, one of which is shown as segment  130 . DDP/RDMAP prefixes each segment  130  with a DDP/RDMAP header  125 . The header  125  combines RDMA control and DDP fields into a single header. The RDMA control field (not shown separately) specifies the RDMA operations. DDP fields (not shown separately) specify parameters such as the address of the destination buffers and the length of data transfer. 
         [0026]    Especially when network packets may be received out-of-order, a receiver can use markers at fixed, known locations to quickly locate DDP headers. After recovering the DDP header, the receiver may place payload data into its destination buffer. Because each DDP segment is self-contained (in that the header includes a destination buffer address), quick data placement in the presence of out-of-order receive packets becomes feasible. This feature reduces the amount of memory required for buffering data at an adapter. 
         [0027]      FIG. 2  illustrates the interaction between an RDMA network interface card (RNIC)  202  and a host  201 . When a host application  205  is to send data via an RDMA/TCP connection, the application  205  issues a transmit request  210  to the send queue (SQ). The command includes the amount of data to be sent. The RDMA protocol suite  220  is responsible for framing the data as shown in  FIG. 1 , prior to processing by a TCP engine  230 . 
         [0028]    Removing markers from received RDMA packets is equally difficult. Before the RNIC can store the data into the memory locations designated by the host application, the RNIC must remove the MPA markers. 
         [0029]    Insertion and removal of a RDMA CRC (cyclic redundancy code) digest is also difficult to handle efficiently. As shown in  FIG. 1 , every framed protocol data unit (PDU) is appended with a CRC digest  150 . 
         [0030]    The invention described herein is an apparatus that efficiently calculates the locations of MPA markers and RDMA CRCs, and inserts and removes them from a data stream. The invention can be a part of an RNIC, and can be embodied in an integrated circuit therein.  FIG. 3  demonstrates the architecture of an ASIC that supports RDMA according to an embodiment of the invention. The illustrated design facilitates the insertion and removal of MPA markers and CRCs on the DMA paths by a transmit interface (TxIF)  310  and a host interface (HIF) module  320  respectively. The operation of these modules is coordinated in part by commands stored in a processor interface (PIF)  330 . 
       Processor Interface (PIF) 
       [0031]    The processor interface, an embodiment of which is shown in  FIG. 4 , contains a set of command queues. 
         [0032]    TxQ  410  is a command queue for accepting data transmitting requests. 
         [0033]    RDMA_TxCWD  420  is a queue where a protocol processor (one of protocol processors  422 ) inserts commands for directing the TCP engine  425  and the TxIF  310  to frame application data and send out RDMA packets. 
         [0034]    RxQ  430  is an input queue to one or more protocol processors  422 . It keeps indications from the receive interface (RxIF)  440  regarding received packets. 
         [0035]    Protocol processors  422  receive packets via the RxQ  430 . After RDMA processing, the protocol processors  422  add commands to the RDMA_RxCWD queue  450 . These commands guide the DMA engines in host interface (HIF)  320  to move received packet data to designated host memory locations. 
       Host Interface (HIF) 
       [0036]      FIG. 5  shows a block diagram of the host interface module  320 , according to an embodiment of the invention. 
         [0037]    When passing received data to the host, an outbound DMA engine  505  accepts commands from command queue RDMA_RxCWD  450  in the processor interface module  330 . The outbound DMA engine  505  fetches packet data from the packet memory via a packet memory controller  510 . Note that engine  505  is referred to here as an “outbound” engine because it is processing data that is outbound from an RNIC, even though the data is ultimately inbound to a host. 
         [0038]    The outbound DMA engine  505  moves data through a data formatter  515 , which calculates the correct MPA marker locations and removes the markers from the byte stream. In parallel, the data formatter  515  calculates the RDMAP CRC for validation purposes. 
         [0039]    Command words in each entry of the RDMA_RxCWD queue  450  contain the following formation: 
         [0040]    Address of the local (source) buffer, where the received packets are stored, 
         [0041]    Address of the destination (host) buffer, 
         [0042]    Number of bytes to copy from local buffer to host buffer, 
         [0043]    Starting TCP sequence number for the very first byte of source data, 
         [0044]    RDMA initial receive sequence number (rdma_irs). 
         [0045]    The data formatter  515  will take out every data byte whose sequence number equals (rdma_irs+n*512+k), where n is an integer and k ε {0,1,2,3}. In other words, 4-byte MPA markers are inserted at a 512 bytes stride, starting at the sequence number rdma_irs. 
       Transmit Interface (TxIF) 
       [0046]    When the host application is to send data, the host application writes a request to the queue TxQ  410  in PIF  330  (see  FIGS. 3 ,  4 ). Referring to  FIGS. 4 and 6 , the protocol processors  422  will then allocate a buffer TxBuf  610  from a SRAM block  620  in the TxIF  310 . 
         [0047]    The protocol processors  422  then direct the TCP Tx engine  425  to prepare and write TCP/IP headers to the allocated buffer TxBuf  610 , and instruct the inbound DMA engine  520  in HIF  320  to copy outgoing data from the host buffer into TxBuf  610 . The protocol processors  422  also write an RDMA header to the allocated buffer TxBuf  610 . 
         [0048]    In an embodiment of the invention, a transmit packet in the TxBuf  610  has the layout shown on the left in  FIG. 7 . In the illustrated embodiment, each TxBuf is 2 KB in size. When one of the protocol processors  422  issues a transmit request, the protocol processor allocates a TxBuf. The first 256 bytes will be reserved. The TCP Tx engine  425  will fill the first 128-byte region  710  with an ethernet header  715 , an IP header  720 , and a TCP header  725 . The protocol processor will fill the next 128-byte region  730  with RDMA protocol headers  735 . 
         [0049]    In the embodiment shown, packet payload  740  starts at the point that is 256 bytes offset from the beginning of the TxBuf  610 . The destination address of a Tx command word given to the inbound DMA engine  520  in HIF  320  is always set to the address of the TxBuf plus 256. The reason for this offset up is that the TCP Tx engine  425  does not understand RDMA headers. Moreover, the size of RDMA headers is not necessarily a constant. 
         [0050]    After HIF  320  has completed copying data from the source host data buffer into the TxBuf  610 , HIF  320  will signal the transmit control/unload logic  630  of the Tx interface  310 . Tx interface  310  then directs its transmit control/unload logic  630  to read the packet out of the TxBuf  610  into the transmit buffer  640 . Along this path, a data formatter  650  will “pack” the data byte stream, resulting in the format shown on the right of  FIG. 7 . The gap between the TCP header  725  and RDMA headers  735  is removed; likewise the gap between RDMA headers  735  and payload  740  is removed. Furthermore, the data formatter  650  will insert MPA markers into the packet at the right positions (such as marker  745 ), add padding  750  to ensure the packet ends at a word-aligned boundary, calculate and append a CRC  755 , and finally calculate the checksum for the TCP/IP headers. 
         [0051]    This is achieved by the following: 
         [0052]    TCP engine  425  provides the TCP/IP header length information. 
         [0053]    The protocol processor provides the following information:
       the RDMA header length information,   the length of the payload to transfer,   the starting TCP sequence number (sseq) for the very first byte of the outgoing TCP packet,   the initial RDMA send sequence number (rdma_iss).       From the length information, the data formatter  650  can gather necessary data. Starting with the sequence number rdma_iss, data formatter  650  will insert a marker, four bytes in length, at a stride of every 512 bytes. Thus, the very first marker should be inserted after the n-th bytes, where   
 
         [0000]        n= 512−|rdma_iss−(sseq mod 512)| 
         [0059]    Afterwards, a marker is inserted at every 512-byte stride. 
       Conclusion 
       [0060]    While some embodiments of the present invention have been described above, it should be understood that it has been presented by way of examples only and not meant to limit the invention. It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined in the appended claims. Thus, the breadth and scope of the present invention should not be limited by the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.