Scalable data using RDMA and MMIO

To improve upon some of the characteristics of current storage systems in general and block data storage systems in particular, exemplary embodiments combine state-of-the art networking techniques with state-of-the-art data storage elements in a novel way. To accomplish this combination in a highly effective way, it is proposed to combine networking remote direct memory access (RDMA) technique and storage-oriented memory mapped input output (MMIO) technique in a system to provide direct access from a remote storage client to a remote storage system with little to no central processing unit (CPU) intervention of the remote storage server. In some embodiments, this technique may reduce the required CPU intervention on the client side. These reductions of CPU intervention potentially reduce latency while providing performance improvements, and/or providing more data transfer bandwidth and/or throughput and/or more operations per second compared to other systems with equivalent hardware.

BACKGROUND

The disclosure generally relates to accessing remote storage devices and particularly to accessing remote storage devices over Remote Direct Memory Access (RDMA).

2. Description of the Related Art

Storage systems in general and block based storage systems are a key element in modern data centers. These systems are designed to retrieve and store large quantities of data. Effectiveness and usefulness of these systems are based upon several criteria and characteristics. The following are some of the characteristics:

(a) The bandwidth of data transfer or throughput, both for retrieve operations and for store operations. This is also often called transfer speed.

(b) The latency until data transfer commences, both for retrieve operations and store operations.

(c) The number of operations per second achievable in the system, both for retrieve operations and store operations.

(d) The system wide storage space, i.e., the amount of data the system can store.

(e) The size scalability of the system, which governs the ability of changing the size of the storage system during its life cycle. A scalable system is one which permits starting with a small system relative to its maximum size and altering its size with ease, as necessary, with limited penalty. Usually penalties are related to performance degradation or system downtime. Another aspect of this characteristic is linearity in scale vs. cost.

(f) The connectivity scalability of the system, which governs the ability of the system to retain its main characteristics whether accessed by a small number or a large number of clients.

There is an ongoing trend of improvement of storage systems with respect to all of these characteristics. This is a result of an ongoing market need and appreciation for such improvements.

SUMMARY

Exemplary embodiments overcome the above disadvantages and other disadvantages not described above. Also, an exemplary embodiment is not required to overcome the disadvantages described above, and an exemplary embodiment of the present inventive concept may not overcome any of the problems described above.

One or more exemplary embodiments provide a computerized method for retrieving data from a memory mapped input/output (MMIO) enabled storage device of a control server communicatively coupled with a client device through a remote direct memory access (RDMA) enabled network interface controller (rNIC). The method includes receiving from the client device, through a first communication protocol, a first entry for a submission queue, the first entry using a second communication protocol that permits communication between the control server and the MMIO enabled storage device; receiving from the client device, through the first communication protocol, a second entry for a send queue, the second entry using the first communication protocol to send a data block from a memory reserved for the client device to the client device; receiving a trigger for the first entry; and receiving, using the second communication protocol, the data block from the MMIO enabled storage device in response to executing the first entry; and transmitting, using the first communication protocol, the data block to the client device in response to the data block being received from the MMIO enabled storage device.

The first communication protocol may include one from among Infiniband, RDMA over Converged Ethernet (RoCE), and iWARP.

The second communication protocol may be NVMexpress.

The first entry may include an instruction set that when executed by the storage device causes the storage device to read a block of data into the memory reserved for the client device.

The second entry may include an instruction set that when executed causes the data block to be written to a memory in the client device.

The trigger may be a doorbell.

The method may further include executing the first entry for in response to writing to the doorbell.

The method may further include generating an interrupt by the storage device.

The method may further include receiving instructions, at the storage device, to generate the interrupt to invoke the trigger.

The method may further include sending the client device an instruction to generate an interrupt to indicate the first entry was executed.

The method may further include receiving from the client device a poll to determine if the first entry was executed.

The method may further include reading a fixed data block from the storage device into the trigger.

The method may further include receiving the interrupt by a processing element of the control server; and invoking the trigger in response to receiving the interrupt.

According to an aspect of an exemplary embodiment, there is provided a computerized method for storing data in a memory mapped input/output (MMIO) enabled storage device of a control server communicatively coupled with a client device through a remote direct memory access (RDMA) enabled network interface controller (rNIC). The method includes receiving through a first communication protocol from the client device a first data block for storing on the storage device; receiving through the first communication protocol from the client device a first entry for a submission queue, the first entry for communicating using a second communication protocol; receiving through the first communication protocol from the client device a second entry for a send queue to send a second data block from a memory reserved for the client device to the client device over the first communication protocol; and receiving a trigger in response to receiving the second entry, and sending the second data block to the client device using the first communication protocol.

The first communication protocol may include one from among Infiniband, RoCE, and iWARP.

The second communication protocol may include NVMexpress.

The second entry for the submission queue may include an instruction set that when executed by the storage device causes the storage device to write the second data block from the memory reserved for the client device.

The method may further include generating an interrupt by the storage device.

The storage device may receive instructions to generate the interrupt to invoke the trigger.

The method may further include sending the client device an instruction to generate an interrupt to indicate the second entry was executed.

The method may further include receiving from the client device a poll to determine if the second entry was executed.

The method may further include reading a fixed data block from the storage device into the trigger.

The method may further include receiving the interrupt by a processing element of the control server; and invoking the trigger.

According to an aspect of an exemplary embodiment, there is provided a non-transitory computer readable medium having stored thereon instructions that when executed by one or more processing elements perform a method of retrieving data from a memory mapped input/output (MMIO) enabled storage device of a control server communicatively coupled with a client device through a remote direct memory access (RDMA) enabled network interface controller (rNIC). The method includes receiving from the client device, through a first communication protocol, a first entry for a submission queue, the first entry using a second communication protocol that permits communication between the control server and the MMIO enabled storage device; receiving from the client device, through the first communication protocol, a second entry for a send queue, the second entry using the first communication protocol to send a data block from a memory reserved for the client device to the client device; receiving a trigger for the first entry; and receiving, using the second communication protocol, the data block from the MMIO enabled storage device in response to executing the first entry; and transmitting, using the first communication protocol, the data block to the client device in response to the data block being received from the MMIO enabled storage device.

According to an aspect of an exemplary embodiment, there is provided a non-transitory computer readable medium having stored thereon instructions that when executed by one or more processing elements perform a method for storing data in a memory mapped input/output (MMIO) enabled storage device of a control server communicatively coupled with a client device through a remote direct memory access (RDMA) enabled network interface controller (rNIC). The method includes receiving through a first communication protocol from the client device a first data block for storing on the storage device; receiving through the first communication protocol from the client device a first entry for a submission queue, the first entry for communicating using a second communication protocol; receiving through the first communication protocol from the client device a second entry for a send queue to send a second data block from a memory reserved for the client device to the client device over the first communication protocol; and receiving a trigger in response to receiving the second entry, and sending the second data block to the client device using the first communication protocol.

According to an aspect of an exemplary embodiment, there is provided a server including a remote direct memory access (RDMA) enabled network interface controller (rNIC) configured to communicate with a client using a first communication protocol and to communicate with a storage device using a second communication protocol; and a memory configured to store a data block received from the storage device using the second communication protocol. The rNIC is further configured to transmit the data block stored in the memory to the client device using the first communication protocol.

The rNIC may include the memory.

According to an aspect of an exemplary embodiment, there is provided a server that includes a remote direct memory access (RDMA) enabled network interface controller (rNIC) configured to communicate with a client device using a first communication protocol and to communicate with a storage device using a second communication protocol, the storage device including a memory reserved for the client device; a memory configured to store a first data block that is for storing on the storage device, in response to the rNIC receiving the data block from the client device; a submission queue configured to store a first entry in response to receiving the first entry from the client device, the first entry facilitating communication between the rNIC and the storage device using the second communication protocol; a send queue configured to store a second entry in response to receiving the second entry from the client device, the second entry facilitating sending a second data block from the memory reserved for the client device to the client device over the first communication protocol; and a doorbell configured to send the second data block to the client device using the first communication protocol in response to being triggered.

DETAILED DESCRIPTION

To improve upon some of the characteristics of current storage systems in general and block data storage systems in particular, exemplary embodiments combine state-of-the art networking techniques with state-of-the-art data storage elements in a novel way. A networking remote direct memory access (RDMA) technique and a storage-oriented memory mapped input output (MMIO) technique are combined in a system in order to provide direct access from a remote storage client to a remote storage device with little to no central processing unit (CPU) intervention of a remote storage server. In some exemplary embodiments, this technique may reduce the required CPU intervention on the client side. These reductions of CPU intervention reduce latency while providing performance improvements, and/or providing more data transfer bandwidth and/or throughput and/or more operations per second compared to other systems with equivalent hardware.

FIG. 1is a non-limiting exemplary schematic illustration of a storage client device100retrieving data from a remote storage device130. The remote storage device130is controlled by a storage server120. The storage client100includes a driver software module102that generates a remote direct memory access (RDMA) write operation. The driver software module102is invoked by request to read a specific storage block into a memory104of the storage client device100. In some embodiments, to perform storage retrieve operations, the driver software module102translates a logical address of the storage block into a physical address on the remote storage device130, which is controlled by the storage server120. The storage client device100uses the driver software module102to issue the RDMA-write operation. The RDMA-write operation writes an entry for a submission queue132dedicated to the storage client into a memory of the storage device130. The entry for a submission queue132is an instruction to write a requested data block50from the storage device130into a send queue122on a memory of a Network Interface Card (NIC) (not shown) of the server120. The NIC is typically and RDMA capable NIC (rNIC). The entry for the submission queue132includes another instruction instructing to write an interrupt into a doorbell124of the NIC that is dedicated to this storage client100and that triggers the data block50stored in the send queue122to be transmitted to the storage client100.

The client100and the server120communicate using a first type of communication protocol. This first type of communication protocol may be Infiniband (IB). Infiniband is a communication protocol for high performance computing offering high throughput and low latency within a computer network. It should be understood that IB is presented here for pedagogical purposes, and it is clear that other communication protocols offering similar functionality, such as RDMA over Converged Ethernet (RoCE), and iWARP, may be used without departing from the scope of this disclosure.

The storage server120and the storage device130communicate using a second type of communication protocol. This second type of communication protocol may be NVMexpress. NVMexpress is a communication protocol for accessing solid-state drives attached through a PCI Express bus. NVMexpress is presented here for pedagogical purposes, and it is clear that other communication protocols offering similar functionality may be used without departing from the scope of this disclosure. The entry for the submission queue132may contain an instruction that causes the storage device130to write an instruction that causes the storage device130to read a requested data block50from a memory134of the storage device130that is reserved for the storage client100. The data block50is at the translated physical address of the memory134, which corresponds to the logical address of the specific storage block of the storage client device100. The entry for the send queue122contains a request for an RDMA-write, which when executed writes the requested data block50to an area of the client memory104that is specified by the driver software module2. Upon completion of the RDMA-write into the doorbell for the submission queue124, an input/output (I/O) operation defined the by submission queue entry is triggered on the storage server120.

Once the I/O operation is executed and completed, the memory area on the storage server120(i.e., the memory on the NIC of the storage server120) contains the requested data block50. In some embodiments, in conformance with the NVMexpress specification, the completion of the I/O operation triggers a message signaled interrupt (MSI), such as MSI-X function. An MSI-X function writes some payload or data into a memory address. In typical usage of MSI-X in general and for NVMexpress in particular, the MSI-X function is programmed to generate an interrupt, by having it write into the address of the local advanced programmable interrupt controller (APIC) of a processing element of the storage server, such as a central processing unit (CPU) with the payload or data being an interrupt vector number. In an embodiment, the MSI-X function is used in an atypical way by instructing a write of the interrupt into a doorbell of the send queue124. The data written by the MSI function will trigger send queue activity, as it rings the doorbell of the send queue124. Upon writing the MSI payload to the doorbell of the send queue124, an RDMA-write over the network (e.g., IB network) is performed in order to transmit the requested data block50from the memory of the NIC to the memory area104on the client100where the driver software module102has instructed to place the data50. Upon completion of the RDMA-write from the storage server120to the storage client100, an interrupt may be triggered on the storage client100, allowing the driver102to complete the operation.

In an alternative embodiment, the driver102may poll for completion.

In another embodiment, the doorbell of the send queue124on the storage server120is filled by reading a fixed block of data from the storage device130into the doorbell124. This embodiment may be slower in general, but is potentially useful for cases where there are more clients than available MSI-X interrupts, typically over 2,048 (two thousand and forty eight).

In yet another embodiment the MSI-X interrupt is received by the APIC. The CPU fills the doorbell124upon reception of the interrupt.

FIG. 2is a non-limiting exemplary schematic illustration of a storage client device100writing data to a remote storage130controlled by a storage server120in accordance with an embodiment. A driver software module102executed on the storage client device100is invoked with a write request for a specific storage block from a memory104of the client device100. The driver software module102translates the logical address of the specific storage block into a physical address on the remote storage device130of the remote storage server120. The driver software module102issues an RDMA-write operation. The RDMA-write operation writes a data block60to be stored into the storage device130from a memory mapped address bus of the storage server120.

The RDMA-write operation also writes an entry for a submission queue132dedicated to this storage client100into a memory134of the storage device130. The entry for a submission queue132is an instruction to write the requested data block60into the storage device130and an instruction instructing to write an interrupt into a doorbell124that is dedicated to this storage client100. The RDMA write also writes an entry for a send queue122for communicating from the storage server120to the storage client device100.

The entry for the submission queue132contains instructions that when executed by the storage device130cause the storage device130to generate a ‘write’ to the storage device130of the stored data block60at the translated area from a memory area reserved for this storage client100. The entry for the send queue122may contain a request for a dummy message from the storage client device100to the server120that is used for communicating the completion of the store operation from the server120to the client100. Upon completion of the RDMA-write into the doorbell for the submission queue124, an I/O operation is triggered on the storage server120. The I/O operation is defined by the entry written to the submission queue132. Upon completion of the I/O operation on the storage device130, the data block from the memory mapped I/O area on the server has been written to the appropriate storage device at the appropriate address. In one embodiment, in conformance with the specification, the completion of the I/O operation triggers an MSI-X function. An MSI-X function writes some payload or data into a memory address. In typical usage of MSI-X in general and for NVMexpress in particular, the MSI-X function is programmed to generate an interrupt, by having it write into the address of the local APIC of a CPU with the payload or data being the interrupt vector number. In an embodiment, the MSI-X function is used in an atypical way by writing into the doorbell of the send queue124. The data written triggers send queue activity. In some embodiments, upon the write to the doorbell of the send queue, which is the MSI payload, an Infiniband ‘send’ is performed over the network (e.g., IB network) transmitting the completion of the store operation to the storage client. In some embodiments, upon the completion of the Infiniband ‘send’ from the storage server120to the storage client device100, an interrupt is triggered on the storage client100so the driver102can complete the operation. In an alternative embodiment, the driver software module102polls for completion of the operation. In another embodiment, the doorbell of the send queue124on the storage server130is filled by reading a fixed block of data from the storage device into the doorbell124. This embodiment may be slower in general, but is potentially useful for cases where there are more clients than available MSI-X interrupts, typically 11248. In yet another embodiment the MSI-X interrupt received by the APIC. The CPU fills the doorbell124upon reception of the interrupt.

FIG. 3is an exemplary non-limiting schematic illustration of an RDMA network70with a plurality of client devices100-1to100-naccessing a plurality of storage devices130-1to130-nthrough a storage server120, in accordance with an exemplary embodiment. The RDMA network70is configured to provide connectivity of various sorts, as may be necessary, including but not limited to, wired and/or wireless connectivity, including, for example, local area network (LAN), wide area network (WAN), metro area network (MAN), worldwide web (WWW), Internet, and any combination thereof, as well as cellular connectivity. In an embodiment, the storage devices130are solid state drive (SSD) appliances. In another embodiment, storage clients100are cloud servers. In some embodiments, the switched network70is an RDMA-enabled Infiniband network. In such embodiments, storage server120and storage clients100have RDMA-enabled Infiniband network interfaces such remote direct memory access (RDMA) enabled network interface controller (rNIC).

In some exemplary embodiments, there are multiple storage servers that contain multiple storage devices. In some embodiments the, storage devices conform to the NVMexpress specification. A storage device may be a block data storage.

In some embodiments, storage clients may run on virtual machines, and in some embodiments the storage clients may run on physical machines. If the storage client devices run on a virtual machine, in some embodiments the underlying physical machine optionally support the single root I/O virtualization (SR-IOV) specification enabling generation of Infiniband RDMA operations from the virtual machine without hypervisor invocation.

In some embodiments, a driver software module102provides the functionality required for the storage system to appear as a blocked data storage element. The driver102and the storage server communicate with a management software module for the purpose of defining the partitioning of the blocked data storage for multiple clients.

Using the methods detailed herein, storage replication is possible for high availability purposes across storage servers and storage devices. A storage client device100may issue a plurality of store and a corresponding plurality of retrieve operations as may be necessary. A first storage server may be the storage client device for a second storage server, in some embodiments.

In an embodiment, a plurality of physical storage devices from a plurality of storage servers include a single logical storage device from the storage client device perspective. In another embodiment, the entire storage area including the aggregation of storage area from a plurality of storage devices on a plurality of storage servers can be divided into a plurality of logical storage devices from the storage client device perspective. In some embodiments, the division of part or all of the entire storage area need not be related to the actual physical devices. In some embodiments, there can be a performance benefit in a division of part of or of the entire storage area that is orthogonal to the actual layout, one which aggregates data from multiple physical devices to implement one or more logical storage devices from the client device perspective. In an embodiment, there is a software module for managing the division of the aggregate storage area into partitions. This software module may use a database to store the partitioning definition. The database may store the partitioning definition in the storage area, in a dedicated management partition. The partitioning of the storage area can be changed online without stopping or rebooting the system. Each storage client may view its own segment or partition of the aggregate storage area. Alternatively, a plurality of clients may perform retrieve or store operations on shared or overlapping segments or partitions of the storage area.

In an embodiment, the number of storage servers and the number of storage devices connected to them or embedded within are constant and the number of storage servers and the number of storage devices cannot be changed during the life cycle of the system, although they may be replaced.

In another embodiment, additional storage servers may be added to the system. Likewise, additional storage devices can be added to storage servers. In an embodiment of the system additional storage, additional storage servers, additional storage devices, or both may optionally be added to storage available to one or more clients.

In one embodiment, one or more of the following additional functionalities may optionally be layered on top of the storage system and/or as methods run on a client device accessing the storage system.

(a) A file system is run on the storage device or one or more of its partitions.

(b) A distributed file system is run on the storage device or one or more of its partitions.

(c) A duplication method is implemented to support failure scenarios where one or more storage elements and/or storage servers fails. Well known methods such as data duplication and error code correction may be employed.

(d) A deduplication method may be implemented to conserve storage area by maintaining fewer copies, perhaps even just one, of data that without this method would have been stored in multiple copies in the storage area.

(e) Security mechanisms that enable using the storage system for different entities with different permissions.

In an embodiment, duplication is optionally embedded into the system to improve the quality of the duplication and to increase improved system availability, i.e. its ability to withstand failures while providing data storage service. Embedding the duplication method into the system may also increase overall performance as combinations of RDMA and MMIO may then be used for this purpose also, as the duplication will occur immediately and require less computation resources on the storage client.

In an embodiment, deduplication is embedded into the system to improve the quality of the deduplication and the reduction in storage area used. Embedding the deduplication method into the system may also increase overall performance as combinations of RDMA and MMIO may then be used for this purpose also.

In an embodiment, some security aspects are embedded into the system to improve the efficiency of the security mechanisms. In an embodiment of the invention, NVMexpress namespaces are used for this purpose, which reduces the required computational resources used to implement the security mechanisms.

The principles herein are implemented as hardware, firmware, software or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium. The application program may be uploaded to, and executed by, a machine including any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as a processing unit (“CPU”), a memory, and input/output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit and/or display unit.

All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure. The specification mentions certain implementations of technologies, such as NVMexpress, Infiniband, and MSI-X. The technology names are provided in an exemplary manner and should be construed as such. All trademarks, service marks, trade names, and product names are the property of their respective owners.