Shared receive queues

The disclosed embodiments relate to a queuing mechanism that may comprise a shared receive queue having a plurality of buffers. The queuing mechanism may also comprise a plurality of queue pairs, each of the plurality of queue pairs having a receive queue that comprises at least one of the plurality of buffers.

BACKGROUND OF THE RELATED ART

In the field of computer systems, it may be desirable for information to be transferred from a system memory associated with one computer system to a system memory associated with another computer system. Queue pairs (“QPs”) may be used to facilitate such a transfer of data. Each QP may include a send queue (“SQ”) and a receive queue (“RQ”) that may be utilized in transferring data from the memory of one device to the memory of another device. The QP may be defined to utilize an allocated number of memory blocks or buffers for each RQ and SQ.

The allocation of specific number of buffers for each SQ and RQ may be inefficient if some RQs and SQs are idle. This situation may occur frequently in a multi-client computing environment that supports numerous QPs. As a result of these inefficiencies; overall system performance may be degraded.

DESCRIPTION OF SPECIFIC EMBODIMENTS

The Remote Direct Memory Access (“RDMA”) Consortium, which includes the assignee of the present invention, is developing specifications to improve ability of computer systems to remotely access the memory of other computer systems. One such specification under development is the RDMA Consortium Protocols Verb specification, which is hereby incorporated by reference. The verbs defined by this specification may correspond to commands or actions that may form a command interface for data transfers between memories in computer systems, including the formation and management of queue pairs, memory windows, protection domains and the like.

RDMA may refer to the ability of one computer to directly place information in the memory space of another computer, while minimizing demands on the central processing unit (“CPU”) and memory bus. In an RDMA system, an RDMA layer may interoperate over any physical layer in a Local Area Network (“LAN”), Server Area Network (“SAN”), Metropolitan Area Network (“MAN”), or Wide Area Network (“WAN”).

Referring now toFIG. 1, a block diagram illustrating a computer network in accordance with embodiments of the present invention is illustrated. The computer network is indicated by the reference numeral100and may comprise a first processor node102and a second processor node110, which may be connected to a plurality of I/O devices126,130,134, and138via a switch network118. Each of the I/O devices126,130,134and138may utilize a Remote Direct Memory Access-enabled Network Interface Card (“RNIC”) to communicate with the other systems. InFIG. 1, the RNICs associated with the I/O devices126,130,134andl38are identified by the reference numerals124,128,132and136, respectively. The I/O devices126,130,134, and138may access the memory space of other RDMA-enabled devices via their respective RNICs and the switch network118.

The topology of the network100is for purposes of illustration only. Those of ordinary skill in the art will appreciate that the topology of the network100may take on a variety of forms based on a wide range of design considerations. Additionally, NICs that operate according to other protocols, such as InfiniBand, may be employed in networks that employ such protocols for data transfer.

The first processor node102may include a CPU104, a memory106, and an RNIC108. Although only one CPU104is illustrated in the processor node102, those of ordinary skill in the art will appreciate that multiple CPUs may be included therein. The CPU104may be connected to the memory106and the RNIC108over an internal bus or connection. The memory106may be utilized to store information for use by the CPU104, the RNIC108or other systems or devices. The memory106may include various types of memory such as Static Random Access Memory (“SRAM”) or Dynamic Random Access Memory (“DRAM”).

The second processor node110may include a CPU112, a memory114, and an RNIC116. Although only one CPU112is illustrated in the processor node110, those of ordinary skill in the art will appreciate that multiple CPUs may be included therein. The CPU112, which may include a plurality of processors, may be connected to the memory114and the RNIC116over an internal bus or connection. The memory114may be utilized to store information for use by the CPU112, the RNIC116or other systems or devices. The memory114may utilize various types of memory such as SRAM or DRAM.

The switch network118may include any combination of hubs, switches, routers and the like. InFIG. 1, the switch network118comprises switches120A–120C. The switch120A connects to the switch120B, the RNIC108of the first processor node102, the RNIC124of the I/O device126and the RNIC128of the I/O device130. In addition to its connection to the switch120A, the switch120B connects to the switch120C and the RNIC132of the I/O device134. In addition to its connection to the switch120B, the switch120C connects to the RNIC116of the second processor node110and the RNIC136of the I/O device138.

Each of the processor nodes102and110and the I/O devices126,130,134, and138may be given equal priority and the same access to the memory106or114. In addition, the memories may be accessible by remote devices such as the I/O devices126,130,134and138via the switch network118. The first processor node102, the second processor node110and the I/O devices126,130,134and138may exchange information using queue pairs (“QPs”). The exchange of information using QPs is explained with reference toFIG. 2.

FIG. 2is a block diagram that illustrates the use of a queue pair to transfer data between devices in accordance with embodiments of the present invention. The figure is generally referred to by the reference numeral200. InFIG. 2, a first node202and a second node204may exchange information using a QP. The first node202and second node204may correspond to any two of the first processor node102, the second processor node110or the I/O devices126,130,134and138(FIG. 1). As set forth above with respect toFIG. 1, any of these devices may exchange information in an RDMA environment.

The first node202may include a first consumer206, which may interact with an RNIC208. The first consumer206may comprise a software process that may interact with various components of the RNIC208. The RNIC208, may correspond to one of the RNICs108,116,126,130,134or138(FIG. 1), depending on which of devices associated with those RNICs is participating in the data transfer. The RNIC208may comprise a send queue210, a receive queue212, a completion queue (“CQ”)214, a memory translation and protection table (“TPT”)216, a memory217and a QP context218.

The second node204may include a second consumer220, which may interact with an RNIC222. The second consumer220may comprise a software process that may interact with various components of the RNIC222. The RNIC222, may correspond to one of the RNICs108,116,126,130,134or138(FIG. 1), depending on which of devices associated with those RNICs is participating in the data transfer. The RNIC222may comprise a send queue224, a receive queue226, a completion queue228, a TPT230, a memory234and a QP context232.

The memories217and234may be registered to different processes, each of which may correspond to the consumers206and220. The queues210,212,214,224,226, or228may be used to transmit and receive various verbs or commands, such as control operations or transfer operations. The completion queue214or228may store information regarding the sending status of items on the send queue210or224and receiving status of items on the receive queue (“RQ”)212or226. The TPT216or230may comprise a simple table or an array of page specifiers that may include a variety of configuration information in relation to the memories217or234.

The QP associated with the RNIC208may comprise the send queue210and the receive queue212. The QP associated with the RNIC222may comprise the send queue224and the receive queue226. The arrows between the send queue210and the receive queue226and between the send queue224and the receive queue212indicate the flow of data or information therebetween. Before communication between the RNICs208and222(and their associated QPs) may occur, the QPs may be established and configured by an exchange of commands or verbs between the RNIC208and the RNIC222. The creation of the QP may be initiated by the first consumer206or the second consumer220, depending on which consumer desires to transfer data to or retrieve data from the other consumer.

Information relating to the configuration of the QPs may be stored in the QP context218of the RNIC208and the QP context232of the RNIC222. For instance, the QP context218or232may include information relating to a protection domain (“PD”), access rights, send queue information, receive queue information, completion queue information, or information about a local port connected to the QP and/or remote port connected to the QP. However, it should be appreciated that the RNIC208or222may include multiple QPs that support different consumers with the QPs being associated with one of a number of CQs.

To prevent interferences in the memories217or234, the memories217or234may be divided into memory regions (“MRs”), which may contain memory windows (“MWs”). An entry in the TPT216or230may describe the memory regions and may include a virtual to physical mapping of a portion of the address space allocated to a process. These memory regions may be registered with the associated RNIC and the operating system. The nodes202and204may send a unique steering tag (“STag”) to identify the memory to be accessed, which may correspond to the memory region or memory window.

The STag may be used to identify a buffer that is being referenced for a given data transfer. A tagged offset (“TO”) may be associated with the STag and may correspond to an offset into the associated buffer. Alternatively, a transfer may be identified by a queue number, a message sequence number, and/or message offset. The queue number may be a 32-bit field, which identifies the queue being referenced. The message sequence number may be a 32-bit field that may be used as a sequence number for a communication, while the message offset may be a 32-bit field offset from the start of the message.

Also, the node202or204may have a unique QP identity for communications with the other node202or204. By using QP, the access to the memory regions and memory windows by the node202or204over the designated QP may be enabled for QPs having the same PD. Each of the RQs212and226for the respective QPs may include buffers that are dedicated to that RQ and be allocated from the memory217or234. These buffers may be blocks of memory that are allocated when the RQs212and226are created. Accordingly, it may be beneficial for the RQs212and226to share buffers across multiple QPs. As such, the buffers may be allocated to a shared receive queue and allocated when a request is received. Thus, the plurality of shared buffers may be utilized to allow the RQs212and226for various QPs to pool resources to enhance the operation of the node. In this manner, RQs212and226may avoid dropping connections when the buffers are pre-allocated to different processes that are not efficiently utilizing them. The interaction between QPs, RQs, SQs, in the context of data transfers employing a queuing mechanism or shared receive queue (“S-RQ”) with multiple QPs is explained with reference toFIG. 3.

FIG. 3is a block diagram illustrating data exchange using a shared receive queue with multiple queue pairs in accordance with embodiments of the present invention. The diagram is generally referred to by the reference numeral300. A consumer308may operate processes, upper layer protocols, or applications on a node302, which may correspond to one of the nodes202or204(FIG. 2). The node302may include a first send queue310and a second send queue311, which may correspond to the send queues210and224ofFIG. 2. Additionally, a first receive queue312and a second receive queue313may be associated with each of the respective receive queues212and226ofFIG. 2. The use of two sets of send queues and receive queues indicates that two sets of QPs have been established for communication between the server node302and other devices. The send queue310and the receive queue312together form a QP that is identified by the reference numeral315. The send queue311and the receive queue313together form a QP that is identified by the reference numeral317.

The QP315may be adapted to exchange information with a corresponding QP323, which may comprise a send queue320and a receive queue322. The QP323may be located in a node304, which may correspond to a device with which the server node302is exchanging information. The arrows between the send queue310and the receive queue322and between the send queue320and the receive queue312indicate the flow of information therebetween. Similarly, the QP317may be adapted to exchange information with a corresponding QP327, which may comprise a send queue324and a receive queue326. The QP327may be located in a node306, which may correspond to a device with which the server node302is exchanging information. The arrows between the send queue311and the receive queue326and between the send queue324and the receive queue313indicate the flow of information therebetween.

The receive queues312and313may be associated with a queuing mechanism or shared receive queue (“S-RQ”)314. When messages directed to the receive queues312and313are received, the request for buffers to place the message may be redirected to the S-RQ314. The S-RQ314may be a located in a memory318, which may be located anywhere within the node302. The S-RQ314may include a group of buffers that may be created by a verb or command, at initialization of the node302or other suitable time. The buffers may be contiguous blocks of memory that are utilized by the RQs312,313. Accordingly, the S-RQ314may share a group of buffers with various RQs based on various parameters, such as a common protection domain for a specific consumer. The size of the S-RQ314may be set by the consumer308and may be modified by limitations or other verbs or commands to maintain operation.

A buffer manager316, which may manage the operation of the S-RQ314, may assign buffers to the RQs when requested by a consumer or when a request is received, such as a work request (“WR”), an incoming RDMA read or write request, or send with invalidate, send with solicited event, send with solicited event and invalidate, or any other similar request. A consumer interface319may be used to process incoming requests from the consumer308, such as when completion of the incoming data has been determined. In response to requests received from the consumer308via the consumer interface319, the buffer manager316may act to limit the number of buffers associated with the RQ312or313in the S-RQ314. Requests to the buffer manager316may also dictate the total number of buffers to associate with the S-RQ314.

The S-RQ314may be implemented and managed through the use of verbs or commands. For instance, a “Create S-RQ” verb may be issued to establish the S-RQ314. A “Modify S-RQ” verb may be used to modify the characteristics of the S-RQ314, such as the number of buffers associated with a particular receive queue. A “Destroy S-RQ” verb may be used to remove the S-RQ314, when the associated QPs have completed their data transmissions. Those of ordinary skill in the art will appreciate that other verbs or commands may be devised for the management of the S-RQ314.

Verbs or commands used in the creation and maintenance of QPs may also be used to impact the S-RQ314. For example, a “Create QP” verb or command may indicate that the S-RQ314is to be utilized by the QP. The indication may involve a setting within verb or command or an associated argument. Further, a “Poll CQ” verb or command may include additional output identifiers. The output identifiers may be used to communicate information about the structure and operation of the S-RQ314to the consumer308.

A data transfer operation to an anonymous buffer may be initiated by a work request with a message. The message may be a send type message, an RDMA read type message, an RDMA write type message, or other similar message. If a message, such as a send type message, is directed to a specific QP, then the message may be directed to a receive buffer that is in the S-RQ314as a work queue entry (“WQE”). The posting of the message as a WQE may include a list of memory locations, such as memory windows or memory regions, from which data is intended to be read or written. The receive buffer may be posted to the RQ312or313from the S-RQ314depending on the appropriate QP associated with the message. The receive buffers pointed to by the WQEs may be removed from the S-RQ314in an implementation specific order that may be unique for each S-RQ314. The protection domain associated with a WQE may be validated against protection domain information in the S-RQ314to make sure the operation is authorized. Accordingly, the S-RQ314may be accessed in any order with respect to the S-RQ314, but may preserve the order for an individual receive queue or associated send queue. When the message represented by a WQE is completed, the completion may be posted to the completion queue of the affected QP.

In an exemplary operation of the S-RQ314in the node302, the nodes304and306may send requests to access the memory318or work requests may be generated by the consumer308. The RQs312and313may be associated with a protection domain that is associated with the S-RQ314, the respective QP, the S-RQ314and the associated QP, or other suitable components. For example, if RQs312and313are associated with a protection domain of the S-RQ314, any RQ associated with the protection domain may utilize the S-RQ314and any validation for the S-RQ314may verify the protection domain in the S-RQ314. Requests may result in the allocation of S-RQ buffers to the various RQs312and314. For instance, if a request is received on QP315, a buffer R1A in the S-RQ314may be allocated to the RQ312. Similarly, if a request is received on QP317, a buffer R2A in the S-RQ314may be allocated to RQ313. If another request is received on QP315, another buffer R1B in the S-RQ314may be allocated to the RQ312. When the respective data transfers are completed, the buffers R1A, R1B and R2A may be reallocated from the RQs312and313to the S-RQ314.

Advantageously, the S-RQ314may reduce the dependence on an upper level communication protocol to provide flow control of information delivered to RQs. Instead of relying on an upper level protocol of the consumer308, flow control of incoming messages may be provided by the buffer manager316by locally handling asynchronous events. The buffer manager316may effectively provide flow control over multiple communication channels (QPs), which share RQs via the S-RQ314. This means that adapting to the changing buffer requirements between the QPs may be faster. Accordingly, the S-RQ314may improve response time for adjusting buffers across multiple QPs.

Various error semantics may be implemented to address errors in the operation of the S-RQ314. For instance, errors relating to a specific QP may be reported through the completion of the WQE in a manner that is non-interruptive. If the S-RQ314fails catastrophically, each of the QPs associated with the S-RQ314may be flushed.

One error that may occur is the out of order receipt of a request. One approach to process out of order requests is to have a sufficiently large number of buffers from the S-RQ314posted to the RQ312or313. If not enough buffers are available, however, a connection may be dropped. Other approaches to processing out of order packets may involve dropping the request that is out of order, dropping subsequent requests that are prior to the out of order packet, pausing the QP processing or the like. As appreciated by those in the art, the approach implemented may vary depending on design preferences.

FIG. 4is a process flow diagram showing the operation of a shared receive queue in accordance with embodiments of the present invention. In the diagram, generally referred to by reference numeral400, a shared receive queue may be implemented and may be utilized in a node, such as the node302(FIG. 3). The shared receive queue or S-RQ may correspond to the S-RQ314(FIG. 3). The process begins at block402. At block404, an S-RQ may be created within a memory device associated with the node. As set forth above, the S-RQ may be created automatically upon initialization of the node or created by the execution of a verb or command.

As shown in block406, various QPs, such as QP312and313(FIG. 3) may be associated with the S-RQ. The QPs may be associated with the S-RQ when each of the RQs is created and the association may be based on a protection domain or other factors.

At block408, a request, such as a work request or an RDMA read orwrite request, may be received for processing by the node. The request may be directed to a QP that is associated with the S-RQ. When the request is received, the request may be validated through various processes. If the request is validated (block410), a buffer may be allocated from the S-RQ to the RQ that corresponds to that request, as shown at block412. Then at block414, the request may continue further processing using the S-RQ that was created. The processing of the request may involve accessing a memory segment, executing a command or the like. However, if the request cannot be validated (block410), then a response message may be generated at block416. The response to the request may include terminating the connection or sending an invalid request message. Accordingly, the process ends at block418.