Abstract:
Provided are a method, system, and program for transferring data between an initiator node and target node. A request is received conforming to a first data transfer protocol at the initiator node to transmit to the target node. A reference to a memory location is obtained to use to transfer the request to the target node. At least one function is called that executes in a user address space of the initiator node, wherein the initiator node includes a kernel address space and the user address space. The at least one function executing in the user address space interfaces with an adaptor to transmit the request and reference to the memory location to the target node using a second data transfer protocol.

Description:
BACKGROUND OF THE INVENTION  
       [0001]     1. Field of the Invention  
         [0002]     The present invention relates to a method, system, and program for executing data transfer requests.  
         [0003]     2. Description of the Related Art  
         [0004]     In storage environments, data access commands are communicated from a host system to a storage controller, which manages access to the disks. The storage controller may be a card inside the host system or a separate device. The Internet Small Computer Systems Interface (iSCSI) protocol is used for storage networks that utilize Ethernet connections, including Ethernet switches and routers. The term “iSCSI” as used herein refers to the syntax and semantic of the iSCSI protocol defined by the IETF (Internet Engineering Task Force) standards body, and any variant of that protocol. In current storage networks where iSCSI is utilized, the packet configuration comprises an Ethernet package encapsulating an Internet Protocol (IP) and Transmission Control Protocol (TCP) package layers, which further encapsulate an iSCSI package that includes one or more SCSI commands. The Ethernet protocol provides for link-level error checking as the packets flow from point-to-point on any network segment (link) to determine whether data has been corrupted while passing on a link. In network data transmission operations, an initiator device transmits data or commands over the network to a target device. The TCP/IP package includes an error detection code to perform an end-to-end checking to determine at the opposite end whether the transmitted packet has changed during the transmission as the packet passes through switches and routers. A receiving device detecting an error will send a negative acknowledgment to the sending device to request retransmission of those packets in which errors were detected.  
         [0005]     The Remote Direct Memory Access (RMDA) protocol provides the capability of one computer to directly place information in another computer&#39;s memory with minimal demands on memory bus bandwidth and processor overhead. RDMA over TCP/IP (also known as iWARP) defines the interoperable protocols to support RDMA operations over standard TCP/IP networks. An RDMA Network Interface Card (RNIC) network adaptor card implements the RDMA protocol and performs RDMA operations to transfer data to local and remote memories. Further details of the RDMA protocol are described in the specifications entitled “RDMA Protocol Verbs Specification (Version 1.0)”, published by the RDMA Consortium (April, 2003); “Direct Data Placement over Reliable Transports (Version 1.0)”, published by RDMA Consortium (October 2002); and “Marker PDU Aligned Framing for TCP Specification (Version 1.0)”, published by the RDMA Consortium (October 2002), and which specifications are incorporated herein by reference in their entirety.  
         [0006]     One specification entitled “iSCSI Extensions for RDMA Specification (Version 1.0), by Michael Ko et al., released by the RDMA Consortium (July, 2003), which specification is incorporated herein in its entirety, defines a protocol for providing the RDMA data transfer capabilities to iSCSI by layering iSCSI on top of RDMA.  
         [0007]     Data transfer systems, including the RDMA and others described above, typically require the intervention of kernel mode processes to handle the read and write operations to and from the storage devices. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0008]     Referring now to the drawings in which like reference numbers represent corresponding parts throughout:  
         [0009]      FIG. 1  illustrates an example of network nodes in which embodiments are implemented;  
         [0010]      FIG. 2  illustrates an example of a computing architecture in accordance with the described embodiments;  
         [0011]      FIG. 3  illustrates a packet format in a manner known in the prior art; and  
         [0012]      FIG. 4  and  5  illustrate operations performed to transfer data in accordance with embodiments.  
       SUMMARY  
       [0013]     Provided are a method, system, and program for transferring data between an initiator node and target node. A request is received conforming to a first data transfer protocol at the initiator node to transmit to the target node. A reference to a memory location is obtained to use to transfer the request to the target node. At least one function is called that executes in a user address space of the initiator node, wherein the initiator node includes a kernel address space and the user address space. The at least one function executing in the user address space interfaces with an adaptor to transmit the request and reference to the memory location to the target node using a second data transfer protocol. 
     
    
     DETAILED DESCRIPTION  
       [0014]     In the following description, reference is made to the accompanying drawings which form a part hereof and which illustrate several embodiments of the present invention. It is understood that other embodiments may be utilized and structural and operational changes may be made without departing from the scope of the present invention.  
         [0015]      FIG. 1  illustrates a network computing environment comprised of a plurality of computing nodes  2   a,    2   b  . . .  2   n  that communicate over a network  4 . The network may comprise a Local Area Network (LAN), Wide Area Network (WAN), Storage Area Network (SAN). Alternatively, the nodes may communicate over a bus, such a SCSI bus, etc.  
         [0016]     The nodes  2   a,    2   b  . . .  2   n  in  FIG. 1  may operate as both initiators and targets.  FIG. 2  illustrates components included in a node, such as nodes  2   a,    2   b,    2   c,  to enable communication over the network  4 . A node  2  includes a processor  6 , such as a central processing unit or complex, and an operating system  8 . The operating system  8  provides a user address space  10  and kernel address space  12 , where user applications execute in the user address space  10  and essential operations execute in the kernel address space  12 , such as resource allocation, low-level hardware interfaces, security, etc.  
         [0017]     An application  14  executes in the user space  10 , where the application  14  comprises a user application, such as a database program, server program, etc. To perform an I/O operation, the application  14  would make a call to a SCSI layer  16  to generate a protocol data unit (PDU) comprising the I/O request, which in turn would make a call to an iSCSI layer  18 , which in turn would call a function from the iSER library  20 . Certain functions in the iSER library  20  would call an RNIC driver  22  executing in the kernel space  12 , while others will invoke the RNIC directly from the user space  10 . A RNIC  24  includes an RDMA layer  26  and network layers  28 , such as a TCP layer, IP layer, and Ethernet layer, to package a packet in the transport layer for transmission over the network  4  or unpackage packets received from the network  4 .  
         [0018]     In certain embodiments, to perform I/O operations, the iSER functions called to handle the I/O request needs to register a memory location in memory  30  to use for the I/O operation. The application  14  may register a memory location, which includes memory regions and memory windows. The RNIC  24  may directly access the registered memory location (locally or locally and remotely) in a logically contiguous fashion. A defined memory location, such as a memory region or memory window, is identified by a steering tag (“STag”) created by the RNIC  24  and used to reference the registered memory location. In certain embodiments, a memory region or subset of a memory region referred to as a memory window may be registered, where a separate STag would be associated with each registered memory location (region or window). The RNIC  24  uses the STag to access the referenced memory location. In certain embodiments, the iSER functions in the iSER library would call the RNIC  22  to register the memory regions by calling an RNIC driver  22  executing in the kernel space  12 . The RNIC driver  22  comprises the device driver to interface the operating system  8  with the RNIC adaptor  24 . In response to the call from the function in the iSER library to declare and register a memory location, e.g., memory region or window, the RNIC driver  22  would call the RNIC  24 . The RNIC Driver  22  (also sometimes called the “Verb” layer) along with the RDMA layer  26  in the RNIC  24  pins the memory location to register, such as memory regions (MRs)  32 , and generates an STag for the memory region/window.  
         [0019]     The RNIC  24  RDMA layer  26  maintains a memory translation table  34 , and when registering a memory region/window, would add an entry to the memory translation table  34  identifying the registered memory region and the STag generated to reference that memory region to enable the RNIC  24  to associate the STag with the memory region. The memory translation table  34  may be maintained within buffers in the RNIC  24  or within the memory  30 . The STags would be returned to the iSER functions requesting the registration to use for I/O operations.  
         [0020]     After the RNIC  24  generates and returns STags to the iSER functions handling the I/O operation, the iSER functions may proceed with the I/O operation. The iSER functions would wrap the packet received from the iSCSI layer  18  with header information and the STag received from the RNIC  24  and pass the packet to the RNIC  24  to transfer. The iSER library  20  functions that interface with the RNIC  24  to perform the RDMA data transfer operation, execute in the user address space  10 , thereby bypassing the RNIC driver  22  executing in the kernel space  12 .  
         [0021]     To manage RDMA data transfers, the RNIC  24  maintains a send queue  36 , a receive queue  38 , and a complete queue  40 . The send queue  36  and receive queue  38  comprise the work queues that the RNIC  24  uses to manage RDMA data transfer requests. The complete queue  40  may comprise a sharable queue containing one or more entries having completion entries to provide a single point of completion notification for multiple work queues. The queues  36 ,  38 , and  40  may be allocated by the RNIC  24  in the memory  30  or within buffers in the RNIC  24 .  
         [0022]      FIG. 3  illustrates the format of a transmitted package used with the embodiments. A SCSI command  50  (such as a read or write command), which is generated by the SCSI layer  16 , is encapsulated by the iSCSI layer  18  within an iSCSI protocol data unit (PDU)  52 , which is further encapsulated by the functions in the iSER library  20  with an iSER header  54 . The iSER header  52  would include the STag and information indicating whether the remote node receiving the advertised STag is to read or write to the memory region (window) referenced by the STag and the work queues related to the request. The iSER encapsulated message  54  is further encapsulated in one or more additional network layers  56 , such as a TCP layer, IP layer, and/or Ethernet layer to allow transmission using these additional network protocols. In certain embodiments, the network layers  28  in the RNIC  24  would assemble the iSER encapsulated message  54  within the additional network layers  56 , such as TCP, IP, Ethernet, etc.  
         [0023]      FIGS. 4 and 5  illustrate operations performed within the user space  10  to implement I/O operations. With respect to  FIG. 4 , upon the initiator iSCSI layer  18  receiving (at block  100 ) a SCSI write request, the iSCSI layer calls (at block  102 ) one or more iSERs function from the iSER library  20  to perform an outbound transfer of a SCSI write command including the memory location  30  that contains the data to write to a target node, which may comprise a storage device. The called iSER function calls (at block  104 ) the RNIC driver  22  in the kernel space  12  to obtain STags for memory regions in which an iSCSI PDU including the received SCSI command is included. The STags may have also been taken from a pool of STags previously registered via kernel  12  calls and thereby bypass the kernel  12  on individual commands. The called iSER function uses (at block  106 ) the iSCSI PDU pointers (passed to it by the iSER function invocation) to create a Scatter/Gather List (SGL) made up of STags and offsets that describe the iSCSI header, and optional data, as a part of a “Send” request to the RNIC, in order to bypass the kernel address space  12 . The iSER function  20 , from the user space  10 , calls (at block  108 ) the RNIC RDMA layer  26  to send the composite iSCSI PDU via an RDMA “Send” including an iSER header that advertises (via STags) the memory location  32  to be used by the remote node to extract additional “SCSI write” data, up to the length indicated in the iSCSI PDU. The remote node upon receiving the iSCSI PDU into a memory region in the remote target node, may then process the iSCSI write request using the iSCSI  18  and SCSI  16  layers at the remote target node, which may comprise a storage device. If all the data to be written to the target node was not sent as part of the iSCSI PDU, then the target node will use the STag contained in the iSER header to extract the rest of the data from the initiator node. When called by the iSER function (on the initiator node), the RNIC  24  uses (at block  110 ) the RDMA  26  and network layers  28  to transfer the message payload to the remote node. The iSER layer at the target may use one or more RDMA operations to extract the requested data from the advertised memory region specified by the STag. After the initiator RNIC receives back from the target node an iSCSI PDU containing the status of the I/O operation the RNIC will invalidate any STag identified in the message to be invalidated. The RNIC  24  may then call (at block  112 ) a function in the iSER library  20  to return status contained in an iSCSI PDU, where the called iSER function executes (at block  114 ) in the user space as a non-privileged user to notify the iSCSI layer of the result of sending the message and to invalidate any other associated STags. Further details on the iSCSI/iSER protocol for writing data are found in the publication “iSCSI Extensions for RDMA Specification”, which was referenced above.  
         [0024]     The target node upon receiving the iSER/iSCSI PDU, unpacks the underlying SCSI write. The target SCSI layer  16  will setup the buffers in memory for the data being written by the initiator and then return a request to the iSCSI layer to begin transmitting data. The target iSCSI layer would then package the request into an iSCSI header. The target iSER layer would intercept the iSCSI request from the target for the data to write and by extracting the STag from the iSER header will be able to issue RDMA Read commands that will cause the referenced data to be read from the Initiators memory. In certain embodiments, the target node iSER/RDMA calls may be entirely or partially included in the kernel address space. (Key of invention is initiator has to use the user space for the iSER, iSCSI, and SCSI functions.) After writing all the data from the initiator, the SCSI/iSCSI process on the target builds a Status PDU which describes the completion status of the SCSI operation, and uses iSER to send the PDU to the Initiator via an RDMA “Send-with-Invalidate” message (which contains the STag to be invalidated.)  
         [0025]      FIG. 5  illustrates operations performed at a node to read data from a remote location. Upon the initiator iSCSI layer  18  receiving (at block  150 ) the SCSI read command, the iSCSI layer  18  calls (at block  152 ) the iSER function to send the iSCSI PDU containing the SCSI Read command to the remote node. The called iSER function calls (at block  154 ) the RNIC driver  22  in the kernel space  12  to obtain STags for memory regions  32  in which the iSCSI/iSER PDU headers are to be extracted, and the location into which the “SCSI Read” data, from the remote target node, is to be placed. The STags may have also been taken from a pool of STags previously registered via kernel  12  calls and thereby bypass the kernel address space  12  on individual commands. The called iSER function includes in its iSER header STags which advertise (at block  156 ) the memory region in which the data is to be written by including the STag as part of the iSER header  52  in a packet having the iSCSI PDU containing the read request to cause the target node to write the requested data specified in the SCSI read request packaged within the iSCSI PDU to the memory region  32  referenced by the advertised STag. The initiator RNIC  24  operating in user address space sends (at block  158 ) the message to the target node. The iSER layer at the target node uses (at block  160 ) one or more RDMA writes to transfer the requested data to the advertised memory region in the initiator node referenced in the STag. The iSER/iSCSI/SCSI layer at the target node may execute in the user address space  10  or kernel  12  address space. In response to the iSER layer, the target RNIC RDMA layer performs (at block  162 ) the RDMA writes to transfer the requested data with a message payload specifying the STag that was advertised in the requesting iSCSI/iSER PDU. When receiving data from a RDMA write issued at the target, the initiator RNIC  24  places (at block  164 ) the requested data into the memory region referenced in the STag. The SCSI/iSCSI/iSER target, upon completing the RDMA write operations, which complete the SCSI Read request, invokes (at block  166 ) the target RNIC to issue a send of SCSI completion status via a Send-with-invalidate message, which contains in the RDMA header the STag used in the RDMA write data. The initiator RNIC  24  upon receiving (at block  168 ) the send-with-invalidate from the initiator invalidates (make unusable) the specified STag in the work queues, and then calls (at block  170 ) a function in the initiator&#39;s iSER library  20  in the user address space  10  to deliver the iSCSI PDU which contains the I/O status message which in turn will be delivered by iSCSI layer  18  to the SCSI layer  16  which also indicates that the returned data requested by the read is in memory  30 . Further details on the iSCSI/iSER protocol for reading data are found in the publication “iSCSI Extensions for RDMA Specification”, which was referenced above.  
         [0026]      FIGS. 4 and 5  illustrate how an iSCSI layer may call functions in an iSER library through the user space  10  as a non-privileged user to communicate directly with the RNIC  24  to perform RDMA related operations without having to make calls to the RNIC driver  22  in the kernel space. In certain embodiments, after memory regions are allocated and STags obtained by iSER functions through the kernel space  10 , RDMA read and write operations to implement iSCSI read and write operations are called through the user space  10 , thereby avoiding the need to call the RNIC driver  22  through the kernel space  12 . Avoiding the kernel address space for processing I/O requests reduces burdens on the node processor because calls through the kernel space  12  place greater resource allocation burdens on the system than calls through the user space  10 . Thus, I/O operations proceed through processes executing in the user address space, thereby avoiding the kernel address space.  
         [0027]     In the described operations, the iSCSI layer  18  called functions in the iSER library  20 . In alternative embodiments, other layers, such as the SCSI layer  16  or application  14 , may build their own iSCSI PDUs and call the iSER layer  20  to implement SCSI read and write requests through RDMA calls made through the iSER library  20 . Likewise, some applications may contain the function of the SCSI layer within their own code.  
         [0028]     Further, the iSCSI layer  18  may invoke additional types of RDMA operations than those described above through calls to the iSER library  20  that are made through the user space  10 , without the need to call the RNIC driver  22  executing in the kernel space  12 . Yet further, the RNIC may return status and other information by invoking functions in the iSER library  20  executed in the user space  10 , to return the status and messages to the iSCSI layer  18  without going through the kernel space  12 .  
       Additional Embodiment Details  
       [0029]     The embodiments described herein may be implemented as a method, apparatus or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof. The term “article of manufacture” as used herein refers to code or logic implemented in hardware logic (e.g., an integrated circuit chip, Programmable Gate Array (PGA), Application Specific Integrated Circuit (ASIC), etc.) or a computer readable medium, such as magnetic storage medium (e.g., hard disk drives, floppy disks, tape, etc.), optical storage (CD-ROMs, optical disks, etc.), volatile and non-volatile memory devices (e.g., EEPROMs, ROMs, PROMs, RAMs, DRAMs, SRAMs, firmware, programmable logic, etc.). Code in the computer readable medium is accessed and executed by a processor. The code in which preferred embodiments are implemented may further be accessible through a transmission media or from a file server over a network. In such cases, the article of manufacture in which the code is implemented may comprise a transmission media, such as a network transmission line, wireless transmission media, signals propagating through space, radio waves, infrared signals, etc. Thus, the “article of manufacture” may comprise the medium in which the code is embodied. Additionally, the “article of manufacture” may comprise a combination of hardware and software components in which the code is embodied, processed, and executed. Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope of the present invention, and that the article of manufacture may comprise any information bearing medium known in the art.  
         [0030]     The described operations may be performed by circuitry, where “circuitry” refers to either hardware or software or a combination thereof. The circuitry for performing the operations of the described embodiments may comprise a hardware device, such as an integrated circuit chip, Programmable Gate Array (PGA), Application Specific Integrated Circuit (ASIC), etc. The circuitry may also comprise a processor component, such as an integrated circuit, and code in a computer readable medium, such as memory, wherein the code is executed by the processor to perform the operations of the described embodiments.  
         [0031]     In described embodiments, the iSER layer was used to implement the RDMA transfer capabilities through the user space  10  to perform iSCSI read and write operations. In additional embodiments, the iSCSI layer may invoke the iSER layer to perform RDMA transfer operations to perform operations other than read and write operations, such as for providing status, messages, etc.  
         [0032]     In the described implementations, the physical layer utilized the Ethernet protocol. In alternative implementations, alternative protocols providing link-to-link checksumming/CRC (or other data protection techniques) of the packet may be used instead of Ethernet, such as Serial Advanced Technology Attachment (SATA), Infiniband, serial attached SCSI cable, etc.  
         [0033]     In described implementations, the transport layer comprised the iSCSI protocol. In alternative implementations other protocols known in the art for transmitting I/O commands in packets and providing end-to-end checksumming/CRC (or other data protection techniques) may be used.  
         [0034]     In the described implementations, the packaged I/O commands comprised SCSI commands. In alternative implementations, the commands may be in different I/O command formats than SCSI, such as Advanced Technology Attachment (ATA) commands.  
         [0035]     In described embodiments, the iSCSI layer made calls to the iSER layer to access the RDMA data transfer capabilities through the user space. In additional embodiments, data transfer protocol layers other than iSCSI executing in the user space, such as an application or other data transfer protocols, may call the iSER layer directly to access RDMA data transfer capabilities.  
         [0036]      FIGS. 4 and 5  describe specific operations occurring in a particular order. In alternative implementations, certain operations may be performed in a different order, modified or removed. Moreover, steps may be added to the above described logic and still conform to the described implementations. Further, operations described herein may occur sequentially or certain operations may be processed in parallel. Yet further, operations may be performed by a single processing unit or by distributed processing units.  
         [0037]     The foregoing description of the implementations has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto. The above specification, examples and data provide a complete description of the manufacture and use of the composition of the invention. Since many implementations of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.