Patent Description:
In a distributed data store network architecture, distributed storage nodes can be used as an intermediary when transferring data between a client device and other storage nodes. However, the transfer of data between the client device and destination storage node, multiple data copies are often required and performed at each intermediary device such as a distributed storage node. Additionally, the data transfer between the client device and the destination storage node may also require multiple RDMA data transfers and multiple buffer allocations.

<CIT> discloses subject matter that can be implemented in, among other things, a method that includes pre-registering multiple memory regions for input/output (IO) buffers of a remote direct memory access (RDMA) interface. The method includes receiving a buffer reservation request from a non-system-based user (NSBU) application through an application programming interface (API). The method includes reserving for the NSBU application a first IO buffer. The method includes receiving a request from the NSBU application through the API to access a file in a distributed file system. The method includes receiving data for the file in the first IO buffer from the distributed file system using the RDMA interface or providing data for the file from the first IO buffer to the distributed file system using the RDMA interface. The method includes receiving a request from the NSBU application through the API to free the first IO buffer.

<CIT> discloses a computer-implemented method, system, and article of manufacture for data communication between a requester and a responder in a remote direct memory access (RDMA) network, where each of the requester and the responder is an RDMA-enabled host of the network. The method includes: sending a request for the responder to provide data, where the request includes a mapped steering tag that is obtained by mapping a set of memory buffers of the requester onto a single representation that allows for identifying each of the memory buffers of the set; and receiving the requested data together with the mapped steering tag and assigning the data being received to the memory buffers of the set consistently with the mapping.

Various examples are now described to introduce a selection of concepts in a simplified form, which are further described below in the detailed description. The Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The invention is set out as in the appended claims.

According to a first aspect of the present disclosure, there is provided a computer-implemented method for remote direct memory access (RDMA) by a distributed storage node. The method includes receiving a request for an input/output (I/O) process associated with data, wherein the I/O process comprises a READ or WRITE operation. In response to the request, a memory segment shared between the operating system and a user process running on the distributed storage node is allocated using an operating system driver of the distributed storage node. The user process includes an I/O stack for processing the request. The shared memory segment includes a context memory portion storing context information associated with the I/O stack, a header memory portion storing header information for the I/O process, and a data memory portion for storing the data. The shared memory segment is registered for RDMA access with a target storage node comprising a target RDMA memory. The target RDMA memory includes a header portion and a data portion. An RDMA transfer between the shared memory segment of the distributed storage node and the target storage node is performed to complete the I/O process. During a WRITE operation header information associated with the WRITE operation and stored in the header memory portion of the shared memory segment as well as data stored in the data memory portion of the shared memory segment is transferred to the header portion and the data portion respectively of the target RDMA memory. During a READ operation header information associated with the READ operation and stored in the header memory portion of the shared memory segment is transferred to the header portion of the target RDMA memory and data stored in the data portion of the target RDMA memory is transferred to the data memory portion of the shared memory segment. The shared memory segment is deallocated in response to receiving a status indicator of completion of the RDMA transfer.

In a first implementation form of the method according to the first aspect as such, the context information is associated with one or more of the following: a data caching operation, a data replication operation, and an RDMA transfer operation performed by the I/O stack to complete the I/O process.

In a second implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the context information includes context information of the operating system driver. More specifically, the context information includes callback context information indicating the user process or a function to be executed by the distributed storage node upon completion of the I/O process, and an I/O structure with active block I/O operations associated with the I/O process.

In a third implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the context information includes a scatter/gather (s/g) list of buffers used by the I/O stack.

In a fourth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the header information comprises one or more of the following: a type of data processing operation associated with the I/O process, a logical unit number (LUN) identifying an object within the target storage node for storing or retrieving the data, a logical block address (LBA) identifying a memory offset for the RDMA transfer, a length information associated with a size of the data, and a snap ID identifying a point in time or a time range for performing the RDMA transfer.

In a fifth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the I/O process includes a READ operation received from a host device, and the method further includes communicating using an RDMA module within the I/O stack, the header information to the target storage node, the header information identifying the LUN and the LBA associated with the data requested by the READ operation.

In a sixth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the data requested by the READ operation and the status indicator is received from the target storage node. The status indicator indicates a successful completion of the READ operation. The requested data and the status indicator are forwarded to the host device.

In a seventh implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the I/O process includes a WRITE operation, and the method further includes performing a direct memory access (DMA) operation by the operating system driver, to store the data in the data memory portion of the shared memory segment.

In an eighth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the RDMA transfer of the header information and the data from the shared memory segment to RDMA memory within the target storage node is performed using an RDMA module within the I/O stack. The status indicator is received from the target storage node, where the status indicator indicates a successful completion of the WRITE operation.

In a ninth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the operating system driver includes one or both of a Fiber Channel (FC) driver or an Internet Small Computer Systems Interface (iSCSI) Extension for RDMA (ISER) driver.

According to a second aspect of the present disclosure, there is provided a distributed storage node including a memory that stores instructions and one or more processors in communication with the memory. The one or more processors execute the instructions to receive a request for an input/output (I/O) process associated with data, wherein the I/O process comprises a READ or WRITE operation. In response to the request, a memory segment shared between the operating system and a user process running on the distributed storage node is allocated using an operating system driver of the distributed storage node. The user process includes an I/O stack for processing the request. The shared memory segment includes a context memory portion storing context information associated with the I/O stack, a header memory portion storing header information for the I/O process, and a data memory portion for storing the data. The shared memory segment is registered for RDMA access with a target storage node comprising a target RDMA memory. The target RDMA memory includes a header portion and a data portion. An RDMA transfer is performed between the shared memory segment of the distributed storage node and the target storage node to complete the I/O process. During a WRITE operation header information associated with the WRITE operation and stored in the header memory portion of the shared memory segment as well as data stored in the data memory portion of the shared memory segment is transferred to the header portion and the data portion respectively of the target RDMA memory. During a READ operation header information associated with the READ operation and stored in the header memory portion of the shared memory segment is transferred to the header portion of the target RDMA memory and data stored in the data portion of the target RDMA memory is transferred to the data memory portion of the shared memory segment. The shared memory segment is deallocated in response to receiving a status indicator of completion of the RDMA transfer.

In a first implementation form of the distributed storage node according to the second aspect as such, the I/O process includes a WRITE operation, and the one or more processors execute the instructions to perform a direct memory access (DMA) operation by the operating system driver, to store the data in the data memory portion of the shared memory segment.

In a second implementation form of the distributed storage node according to the second aspect as such or any preceding implementation form of the second aspect, the RDMA transfer of the header information and the data from the shared memory segment to RDMA memory within the target storage node is perform using an RDMA module within the I/O stack. The status indicator is received from the target storage node, where the status indicator indicates a successful completion of the WRITE operation.

In a third implementation form of the distributed storage node according to the second aspect as such or any preceding implementation form of the second aspect, the header information includes one or more of the following: a type of data processing operation associated with the I/O process, a logical unit number (LUN) identifying an object within the target storage node for storing or retrieving the data, a logical block address (LBA) identifying a memory offset for the RDMA transfer, a length information associated with a size of the data, and a snap ID identifying a point in time or a time range for performing the RDMA transfer.

In a fourth implementation form of the distributed storage node according to the second aspect as such or any preceding implementation form of the second aspect, the I/O process includes a READ operation received from a host device, and the one or more processors execute the instructions to communicate using an RDMA module within the I/O stack, the header information to the target storage node. The header information identifies the LUN and the LBA associated with the data requested by the READ operation.

In a fifth implementation form of the distributed storage node according to the second aspect as such or any preceding implementation form of the second aspect, the one or more processors execute the instructions to receive the data requested by the READ operation and the status indicator from the target storage node, the status indicator indicating a successful completion of the READ operation. The requested data and the status indicator are forwarded to the host device.

In a sixth implementation form of the distributed storage node according to the second aspect as such or any preceding implementation form of the second aspect, the context information includes context information of the operating system driver, such as callback context information indicating the user process or a function to be executed by the distributed storage node upon completion of the I/O process, and an I/O structure with active block I/O operations associated with the I/O process.

According to a third aspect of the present disclosure, there is provided a non-transitory computer-readable medium storing instruction for implementing the method according to the first aspect or an implementation form of the first aspect as described further above.

It should be understood at the outset that although an illustrative implementation of one or more embodiments is provided below, the disclosed systems and/or methods described with respect to <FIG> may be implemented using any number of techniques, whether currently known or not yet in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims.

In the following description, reference is made to the accompanying drawings that form a part hereof, and in which are shown, by way of illustration, specific embodiments which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the inventive subject matter, and it is to be understood that other embodiments may be utilized, and that structural, logical, and electrical changes may be made without departing from the scope of the present disclosure. The following description of example embodiments is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

As used herein, the term "network-based service infrastructure" includes a plurality of network devices providing on-demand computing capacity (e.g., via one or more virtual machines or other virtual resources running on the network devices) and storage capacity as a service to a community of end-recipients (e.g., customers of the service infrastructure), where the end recipients are communicatively coupled to the network devices within the service infrastructure via a network. The customers of the service infrastructure can use one or more computing devices (also referred to as customer devices or host devices) to access and manage the services provided by the service infrastructure via the network. The customer devices, the network, and the network-based service infrastructure can be collectively referred to as a "network architecture. " The customers of the service infrastructure can also be referred to as "users.

As used herein, the term "memory access event" (or "MA event") can include a request for a READ operation (e.g., from memory or other type of storage) or a request for a WRITE operation (e.g., to memory or other type of storage), as well as any data associated with such operations (e.g., data for storage in connection with a requested WRITE operation). As used herein, the term "MA event" is synonymous with the term "I/O event" and "I/O process.

Conventional distributed storage systems using RDMA perform multiple memory allocations (e.g., multiple context memory allocations as data is processed by the I/O stack) as well as multiple data copies. For example, in order to perform the RDMA between a first storage node (i.e., an initiator) and a second storage node (i.e., a target), conventional RDMA techniques perform multiple data copies as the target data is moved between the buffer allocated by the frontend driver and the buffer that is registered for RDMA. Conventional distributed storage systems also use multiple data transfers to move the data via RDMA. For example, one data transfer might move data between a user device and a first storage node via a frontend driver such as a Fiber Channel (FC) driver, an iSCSI Extension for RDMA (ISER) driver, or non-volatile memory express (NVMe) over Fabrics (NVMe-oF) driver. A second data transfer can be via RDMA between the first storage node and the second storage node.

In this regard, distributed storage systems that transfer data through a storage node to a host device or another storage node (e.g., data is transferred from a host device to Storage Node A to Storage Node B, or vice versa), performance is reduced due to multiple data copies, multiple RDMA data transfers, and multiple buffer allocations (e.g., at the intermediary node such as Storage Node A). These attributes of the conventional techniques lead to larger overall latency and overall performance degradation.

RDMA techniques disclosed herein use a shared memory segment within a distributed storage node, where the memory segment is shared between a user space and an operating system (OS)/kernel of the node, or between different user processes running within the user space of the node, where the user processes are associated with different user process address spaces.

As used herein, the term "user space" refers to a set of programs or processes that are running outside of the OS/kernel memory address space associated with the OS/kernel of a device. Put another way, the user space includes processes that do not need access to kernel-related functions, with each process using its own user process address space. As used herein, the term "address space" refers to a range of memory addresses used by a particular process. If a process is a user process (i.e. a process that does not need access to the kernel-related functions), then it may only access memory addresses associated with its address space. Whereas, if a process is a kernel process, it may access any address of the kernel.

The shared memory segment includes a single context memory portion that can be shared by multiple processes performed by the I/O stack of the user space (e.g., a frontend module, a cache module, a replication module, and a RDMA module of an I/O stack, as described hereinbelow) as well as one or more kernel processes. The shared memory segment can also include a header memory portion for storing header information, a status memory portion for storing a status of an I/O operation, and a data memory portion for storing data (e.g., data associated with a READ or a WRITE operation). The shared memory segment is registered for RDMA access with RDMA memory at a target storage device.

By using the shared memory segment that is shared between the user space and the OS/kernel of the node, or between different user processes running within the user space of the node, optimal performance is achieved by eliminating extra data copies, achieving zero data copy operation throughout a storage node I/O stack. Additionally, by sharing a single memory allocation for context across the entire I/O lifecycle, extra memory allocations in the I/O lifecycle are not needed since a single shared allocation is used for all context memory requirements. In this regard, techniques disclosed herein can be used for simplified RDMA transactions between a distributed storage node and a remote node by using a fixed allocation of memory with known addresses (e.g., the shared memory segment), which provides for optimal performance. The allocated shared memory segment can be deallocated after confirmation of successful completion of an I/O process, which increases memory utilization efficiency.

<FIG> is a high-level system overview of a network architecture using a distributed storage node with RDMA functionalities, according to some example embodiments. Referring to <FIG>, the network architecture <NUM> can include a plurality of devices (e.g., user devices) 102A,. , 102N (collectively, devices <NUM>) communicatively coupled to a network-based service infrastructure <NUM> via a network <NUM>. The devices 102A,. , 102N are associated with corresponding users 106A,. , 106N and can be configured to interact with the network-based service infrastructure <NUM> using a network access client, such as one of clients 104A,. The network access clients 104A,. , 104N can be implemented as web clients or application (app) clients.

Users 106A,. , 106N may be referred to generically as "a user <NUM>" or collectively as "users <NUM>. " Each user <NUM> may be a human user (e.g., a human being), a machine user (e.g., a computer configured by a software program to interact with the devices <NUM> and the network-based service infrastructure <NUM>), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). The users <NUM> are not part of the network architecture <NUM> but are each associated with one or more of the devices <NUM> and may be users of the devices <NUM> (e.g., the user 106A may be an owner of the device 102A, and the user 106N may be an owner of the device 102N). For example, the device 102A may be a desktop computer, a vehicle computer, a tablet computer, a navigational device, a portable media device, or a smartphone belonging to the user 106A. Users 106A,. , 106N can use devices 102A,. , 102N to access services (e.g., serverless computing services or storage-related services) provided by the network-based service infrastructure <NUM>. The serverless computing services can include instantiating and using virtual machines (VMs), virtual private clouds (VPCs), application containers (e.g., warm containers instantiated within a VPC), and so forth. The storage-related services can include data storage services, data replication services, and so forth.

The network-based service infrastructure <NUM> can include a plurality of computing devices such as storage nodes <NUM>, <NUM>,. At least one of the storage nodes, such as node <NUM>, can be a distributed storage node functioning as an intermediary between the client devices <NUM> and a destination storage node, such as storage node <NUM>, in connection with an MA event initiated by the user <NUM>.

The distributed storage node <NUM> can be referred to as an initiator node or an initiator, and the destination storage node <NUM> can be referred to as a target node or a target. As illustrated in <FIG>, the distributed storage node <NUM> can include an OS/kernel <NUM> that is used for executing OS related functions and processes, and a user space <NUM> for executing programs and processes running outside of the OS memory address space used by the OS <NUM>.

Any of the devices shown in <FIG> may be implemented in a general-purpose computer modified (e.g., configured or programmed) by software to be a special-purpose computer to perform the functions described herein for that machine, database, or device. As used herein, a "database" is a data storage resource that stores data structured as a text file, a table, a spreadsheet, a relational database (e.g., an object-relational database, a NoSQL database, a network or graph database), a triple store, a hierarchical data store, or any suitable combination thereof. Additionally, data accessed (or stored) via an application programming interface (API) or remote procedure call (RPC) may be considered to be accessed from (or stored to) a database. Moreover, any two or more of the devices or databases illustrated in <FIG> may be combined into a single machine, database, or device, and the functions described herein for any single machine, database, or device may be subdivided among multiple machines, databases, or devices.

The network <NUM> may be any network that enables communication between or among machines, databases, and devices (e.g., devices 102A,. , 102N and devices <NUM>, <NUM>,. , <NUM> within the network-based service infrastructure <NUM>). Accordingly, the network <NUM> may be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The network <NUM> may include one or more portions that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof.

In some aspects, one or more of the users <NUM> can communicate memory access events, such as MA event <NUM> to the network-based service infrastructure <NUM> via the network <NUM>. The MA event <NUM> can trigger RDMA related functionalities such as performing a READ or WRITE operations via RDMA between the storage nodes within the infrastructure <NUM>.

In aspects when the MA event <NUM> is for a WRITE operation, processing within the distributed storage node <NUM> is performed as follows. Initially, a kernel driver such as the FC driver <NUM> or the ISER driver <NUM> allocates kernel data memory <NUM> for storing data received from the client devices <NUM> and kernel context memory <NUM> storing context information associated with kernel functionalities. Next, the kernel driver <NUM> or <NUM> will initiate the data transfer with the client device <NUM> (e.g., 102A) to transfer the data from the client device into the data memory <NUM> allocated by the kernel driver. Subsequently, control of the I/O process of the WRITE operation is switched to the distributed storage node user space <NUM>, which picks up the I/O and continues processing. The distributed storage node <NUM> will then allocate context memory <NUM> and RDMA-capable memory (also referred to as RDMA memory) <NUM> within the user space <NUM>.

The context memory <NUM> is used to store the state of the I/O from the point of view of the distributed storage node, and can include multiple context memory portions (e.g., context memory <NUM>,. , context memory N) that are used to store user context as data traverses from one software module to the next within the I/O stack. The RDMA memory <NUM> is used for copying the data from the kernel data memory <NUM> to the RDMA-capable memory <NUM>. The RDMA memory <NUM> is a memory, which has been registered for RDMA with the computing device <NUM>. As the I/O processing of the WRITE operation traverses the I/O stack, typically many other small context-specific allocations will occur as the I/O processing is handled by one or more processes within the user space <NUM>.

At a final stage, the RDMA occurs with the distributed storage node <NUM> transferring the data associated with the WRITE operation from the RDMA memory <NUM> to the RDMA memory <NUM> within the storage node (or target node) <NUM>. In this regard, the RDMA transfer occurs with multiple phases where each phase signifies a round trip transfer of data and/or state from the initiator node (e.g., <NUM>) to the target node (e.g., <NUM>).

In some aspects, the distributed storage node <NUM> can use shared memory <NUM> in connection with RDMA-related functionalities associated with the memory access event <NUM>. Techniques disclosed herein in connection with <FIG> are associated with using shared memory (e.g. <NUM>) for performing RDMA-related functions. By using the shared memory <NUM> that is shared between the user space <NUM> and the OS/kernel <NUM> (or is shared between different user processes running within the user space <NUM>), optimal performance is achieved by eliminating extra data copies, achieving zero data copy operation throughout a storage node I/O stack.

<FIG> is a block diagram <NUM> illustrating a front-end I/O with RDMA using a shared memory segment, according to some example embodiments. Referring to <FIG>, the initiator shared memory <NUM> can be allocated by a kernel driver (e.g., <NUM> or <NUM>) for each I/O associated with an MA event. The initiator shared memory <NUM> can include a context portion <NUM>, a header portion <NUM>, a status portion <NUM>, and a data portion <NUM>.

The context portion <NUM> can be configured to store context information for the kernel <NUM> (e.g., driver related context <NUM>) and for the user space <NUM> associated with a lifecycle of an I/O. Context information examples for the kernel <NUM> can include a bio structure and callback context for user space interaction. A bio structure is a basic container for block I/O within kernel <NUM>, and it represents block I/O operations that are in flight (active) as a list of segments. A segment is a chunk of a buffer that is contiguous and memory.

Typically, when one I/O stack level hands control to another level, a callback function and a callback context are provided. The callback function includes a pointer to a function, and a callback context is a pointer to an application structure or an application object. When a callback function is invoked to return status to the originator, the callback context is provided which contains enough state to restart the I/O operation and, for example, continue the operation or complete the operation. For example, the callback context might have a pointer to memory that needs to be freed and/or it might have another set of callback context to be called when the current I/O stack level is complete. This type of context can be used in example caching or replication operations when data traverses the I/O stack (the I/O stack is illustrated in greater detail in <FIG>).

Example user space context information includes IoView key, scatter-gather (sg) lists, and RDMA work requests. The IoView key is a key which allows a level of the I/O stack to determine to which node or nodes to send the I/O. In this regard, the key can be perceived as a definition of where data is located for a particular logical unit number (LUN). The sg list can be used to specify the set of fixed length buffers to be used to transfer data of a dynamic size. The RDMA work request is a data structure describing an RDMA transfer to be performed.

The header portion <NUM> is used to store information describing the I/O, including LUN, offset information such as a logical block address (LBA), length information associated with data size, and snap ID. The snap ID is an identifier to a particular snap point in time, identifying a time range from which to fetch the data for an RDMA transfer. More specifically, the header information can indicate a type of data processing operation (e.g., a READ or a WRITE operation) associated with the I/O process. The LUN identifies an object within the target node for storing or retrieving the data).

The status portion <NUM> is used to store status information associated with an RDMA transfer (e.g., an RDMA READ or WRITE operation) between the initiator shared memory <NUM> and the target RDMA memory <NUM>. In some aspects, the status information can include a single byte for indicating status. The data portion <NUM> is used to store data associated with an RDMA transfer (e.g., data communicated from a computing device for storage in the target RDMA memory <NUM> in connection with a WRITE operation, or data communicated from the target RDMA memory <NUM> to a client device via the initiator shared memory <NUM> during a READ operation).

The target RDMA memory <NUM> can include a context portion <NUM>, a header portion <NUM>, a status portion <NUM>, and a data portion <NUM>, which portions are similar in functionality to corresponding portions <NUM> - <NUM> of the initiator shared memory <NUM>.

The shared memory <NUM> is registered for RDMA operation on both the initiator and the target side so that information (e.g., header information, status information, and data) can be exchanged via RDMA between the initiator <NUM> and the target <NUM>. Similarly, the target RDMA memory <NUM> is also registered for RDMA operations on both the initiator and the target side.

By using a single allocation of the initiator shared memory <NUM> within the initiator <NUM>, extra memory allocations during the I/O lifecycle can be avoided. Additionally, local direct memory access (DMA) via the FC driver <NUM> or the ISER driver <NUM> (e.g., in connection with DMA data <NUM>) can also use the shared memory <NUM>, which avoids extra data copies and reduces additional memory use. In this regard, RDMA transfers include an optimal number of transfers from an optimal number of buffers (i.e., one RDMA transfer to start an RDMA operation and another transfer to complete the RDMA operation, all using the single shared memory <NUM>).

<FIG> is a block diagram of a storage system <NUM> using a distributed storage node shared memory for performing a WRITE operation with a target storage node via RDMA, according to some example embodiments. Referring to <FIG>, the shared memory <NUM> can be allocated by a kernel driver within the distributed storage node <NUM> as discussed previously. The shared memory <NUM> and the RDMA memory <NUM> are both registered for RDMA operation at nodes <NUM> and <NUM>. As illustrated in <FIG>, the context portion <NUM> of shared memory <NUM> is used for storing driver related context <NUM> of the kernel <NUM> as well as storage node related context <NUM> generated as data is processed by different modules of the I/O stack within the user space <NUM>.

In some aspects, the MA event <NUM> communicated by the client device 102A can indicate a WRITE operation and can further include data <NUM> that is used for performing the WRITE operation upon. <FIG> and <FIG> provide additional information regarding usage of the shared memory <NUM> for performing the WRITE operation.

<FIG> is a block diagram illustrating a communication exchange between the distributed storage node shared memory and the RDMA memory of the target storage node during the WRITE operation of <FIG>, according to some example embodiments. Referring to <FIG> and <FIG>, at operation <NUM>, a direct memory access (DMA) takes place in connection with the WRITE operation of the MA event <NUM> and data <NUM> is stored within the data portion <NUM> of the shared memory <NUM>. At operation <NUM>, header information (stored in the header portion <NUM>) associated with the WRITE operation as well as the data <NUM> stored in the data portion <NUM> of the shared memory <NUM> is transferred from the shared memory <NUM> of the initiator node <NUM> to the header portion <NUM> and the data portion <NUM> respectively of the RDMA memory <NUM> of the target node <NUM>. The target node <NUM> then performs the WRITE operation using the transferred data <NUM> and the header information stored in the header portion <NUM>. At operation <NUM>, status information for the WRITE operation (e.g., indicating completion or error for the WRITE operation) is transferred via DMA from the status portion <NUM> of the RDMA memory <NUM> to the status portion <NUM> of the shared memory <NUM>. In some aspects, the returned status information can include a single byte. As illustrated in <FIG> and <FIG>, a single shared memory allocation is sufficient to follow the entire I/O processing associated with WRITE operation received with the MA event <NUM>, with zero data copies within the initiator node <NUM> and optimal number of RDMA transfers (e.g., only two RDMA transfers of data are necessary for performing operations <NUM> and <NUM>).

<FIG> is a flowchart of a method suitable for performing the WRITE operation of <FIG>, according to some example embodiments. The method <NUM> includes operations <NUM>, <NUM>, and <NUM>. By way of example and not limitation, the method <NUM> is described as being performed by the initiator node <NUM>. At operation <NUM>, a memory access event (e.g., MA event <NUM>) is received at the initiator node <NUM> from the client device 102A. The MA event indicates a WRITE operation and also includes the data <NUM> associated with the WRITE operation. Shared memory <NUM> is allocated and the received data is stored in the data portion of the shared memory. At operation <NUM>, header information (stored in the header portion <NUM>) associated with the WRITE operation as well as the data <NUM> stored in the data portion <NUM> of the shared memory <NUM> is transferred (e.g., via DMA transfer) from the shared memory <NUM> of the initiator node <NUM> to the header portion <NUM> and the data portion <NUM> respectively of the RDMA memory <NUM> of the target node <NUM>. After the WRITE operation is performed using the transferred data <NUM> and the header information stored in the header portion <NUM>, at operation <NUM>, status information for the WRITE operation (e.g., indicating completion or error for the WRITE operation) is received by the initiator node <NUM> via a DMA transfer from the status portion <NUM> of the RDMA memory <NUM> to the status portion <NUM> of the shared memory <NUM>. Upon successful completion of the WRITE operation (e.g., as indicated by the status information), the allocated shared memory <NUM> can be deallocated.

<FIG> is a block diagram of a storage system <NUM> using a distributed storage node shared memory for performing a READ operation with a target storage node via RDMA, according to some example embodiments. Referring to <FIG>, at operation <NUM>, the shared memory <NUM> is allocated by a kernel driver within the distributed storage node <NUM>, such as FC driver <NUM> or ISER driver <NUM> upon receiving the MA event <NUM> indicating a READ operation. The shared memory <NUM> and the RDMA memory <NUM> are both registered for RDMA operation at storage nodes <NUM> and <NUM>. As illustrated in <FIG>, the context portion <NUM> of shared memory <NUM> is used for storing driver related context <NUM> of the kernel <NUM> as well as storage node related context <NUM> generated as data is processed by different modules of the I/O stack within the user space <NUM>.

In some aspects, the MA event <NUM> communicated by the client device 102A can indicate the READ operation and can further identify the data to be obtained during the READ operation. <FIG> and <FIG> provide additional information regarding the use of the shared memory <NUM> for performing the READ operation.

<FIG> is a block diagram illustrating a communication exchange between the distributed storage node shared memory and the RDMA memory of the target storage node during the READ operation of <FIG>, according to some example embodiments. Referring to <FIG> and <FIG>, after the MA event <NUM> with the READ operation is received by the initiator device <NUM> and the shared memory <NUM> is allocated, at operation <NUM>, a DMA takes place and header information for the READ operation is transferred from the header portion <NUM> of shared memory <NUM> to the header portion <NUM> of RDMA memory <NUM> at the target device <NUM>. The target device <NUM> uses the header information (e.g., LUN, offset, LBA, and so forth) to perform the READ operation. The data obtained during the READ operation is stored in the data portion of the RDMA memory <NUM>. At operation <NUM>, status information from the status portion <NUM> (e.g., indicating successful completion of the READ) and the data from the data portion <NUM> of the RDMA memory <NUM> is transferred via a DMA transfer to the status portion <NUM> and the data portion <NUM> respectively of the shared memory <NUM>. At operation <NUM>, the data stored in the data portion <NUM> received in response to the requested READ operation is transferred via DMA transfer back to the client device 102A. In some aspects, the returned status information can include a single byte, indicating successful completion of the READ operation or an error that has occurred during the READ operation. As illustrated in <FIG> and <FIG>, a single shared memory allocation is sufficient to follow the entire I/O processing associated with the READ operation received with the MA event <NUM>, with zero data copies within the initiator node <NUM> and optimal number of RDMA transfers (e.g., only two RDMA transfers of data are necessary for performing operations <NUM> and <NUM>).

<FIG> is a flowchart of a method suitable for performing the READ operation of <FIG>, according to some example embodiments. The method <NUM> includes operations <NUM>, <NUM>, and <NUM>. By way of example and not limitation, the method <NUM> is described as being performed by the target node <NUM>. At operation <NUM>, a DMA takes place and header information for the READ operation is received from the header portion <NUM> of shared memory <NUM> into the header portion <NUM> of the RDMA memory <NUM> at the target device <NUM>. At operation <NUM>, the target device <NUM> uses the header information (e.g., LUN, offset, LBA, and so forth) to perform the READ operation. The data obtained during the READ operation is stored in the data portion of the RDMA memory <NUM>. At operation <NUM>, status information from the status portion <NUM> and the data from the data portion <NUM> of the RDMA memory <NUM> is transferred via a DMA transfer to the status portion <NUM> and the data portion <NUM> respectively, of the shared memory <NUM>. Upon successful completion of the READ operation (e.g., as indicated by the status information), the allocated shared memory <NUM> can be deallocated.

In aspects disclosed in connection with <FIG>, a first set of modules resides within the OS/kernel <NUM> and is associated with a large, flat address space of the kernel. A second set of modules in connection with <FIG> is a set of modules residing within the user space <NUM>. This set of modules resides in a user process which consists of an address space that is particular to that user process, and which address space is different from that of the kernel. In these aspects, the shared memory <NUM> is mapped to both address spaces.

In other aspects as illustrated and discussed in connection with <FIG>, two different sets of modules are illustrated that reside inside different user processes (process <NUM> and process <NUM>), which have different user process address spaces (e.g., <NUM>, <NUM>). In these aspects, the shared memory (e.g., <NUM>) is mapped into both address spaces used by both user processes.

<FIG> is a block diagram illustrating communication exchange between a distributed storage node memory shared between two user processes and RDMA memory of a target storage node during a WRITE operation, according to some example embodiments. Referring to <FIG>, the shared memory <NUM> can be allocated by a driver <NUM> within the distributed storage node <NUM>, with the memory <NUM> being shared between the two user processes (process <NUM> and <NUM>) with corresponding user process address space <NUM> and <NUM>. As illustrated in <FIG>, the driver <NUM> resides inside the user process address space <NUM> and is user space based. The shared memory <NUM> and the RDMA memory <NUM> are both registered for RDMA operation at nodes <NUM> and <NUM>. The memory portions <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> have functions that are similar to the functions of memory portions <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, respectively.

As illustrated in <FIG>, the context portion <NUM> of the shared memory <NUM> is used for storing driver related context <NUM> of the user process address space <NUM> as well as storage node related context <NUM> generated as data is processed by different modules of the I/O stack within the user process address space <NUM>. In some aspects, the MA event <NUM> communicated by the client device 102A can indicate a WRITE operation and can further include data <NUM> that is used for performing the WRITE operation upon.

At operation <NUM>, a direct memory access (DMA) takes place in connection with the WRITE operation of the MA event <NUM> and data <NUM> is stored within the data portion <NUM> of the shared memory <NUM>. At operation <NUM>, header information (stored in the header portion <NUM>) associated with the WRITE operation as well as the data <NUM> stored in the data portion <NUM> of the shared memory <NUM> is transferred from the shared memory <NUM> of the initiator node <NUM> to the header portion <NUM> and the data portion <NUM> respectively of the RDMA memory <NUM> of the target node <NUM>. The target node <NUM> then performs the WRITE operation using the transferred data <NUM> and the header information stored in the header portion <NUM>. At operation <NUM>, status information for the WRITE operation (e.g., indicating completion or error for the WRITE operation) is transferred via DMA from the status portion <NUM> of the RDMA memory <NUM> to the status portion <NUM> of the shared memory <NUM>. Upon successful completion of the WRITE operation, the allocated shared memory <NUM> can be deallocated.

<FIG> is a block diagram illustrating communication exchange between a distributed storage node memory shared between two user processes and RDMA memory of a target storage node during a READ operation, according to some example embodiments. Referring to <FIG>, at operation <NUM>, the shared memory <NUM> is allocated by a driver within the distributed storage node <NUM> upon receiving the MA event <NUM> indicating a READ operation. As illustrated in <FIG>, the context portion <NUM> of shared memory <NUM> is used for storing driver related context <NUM> of the device kernel as well as storage node related context <NUM> generated as data is processed by different modules of the I/O stack within the user space <NUM>.

In some aspects, the MA event <NUM> communicated by the client device 102A can indicate the READ operation and can further identify the data to be obtained during the READ operation. After the MA event <NUM> with the READ operation is received by the initiator device <NUM> and the shared memory <NUM> is allocated, at operation <NUM>, a DMA takes place and header information for the READ operation is transferred from the header portion <NUM> of shared memory <NUM> to the header portion <NUM> of RDMA memory <NUM> at the target device <NUM>. The target device <NUM> uses the header information (e.g., LUN, offset, LBA, and so forth) to perform the READ operation. The data obtained during the READ operation is stored in the data portion of the RDMA memory <NUM>. At operation <NUM>, status information from the status portion <NUM> (e.g., indicating successful completion of the READ) and the data from the data portion <NUM> of the RDMA memory <NUM> is transferred via a DMA transfer to the status portion <NUM> and the data portion <NUM> respectively of the shared memory <NUM>. At operation <NUM>, the data stored in the data portion <NUM> received in response to the requested READ operation is transferred via DMA transfer back to the client device 102A. Upon successful completion of the READ operation (e.g., as indicated by the status information and after the data is communicated to the client device 102A), the shared memory <NUM> can be deallocated.

<FIG> is a block diagram <NUM> illustrating I/O communications stacks that can be used in connection with RDMA access between storage nodes, according to some example embodiments. Referring to <FIG>, storage node <NUM> communicates with storage node <NUM> via a communications link <NUM>, which can include a wired or wireless connection. The MA event <NUM> is received from client device 102A by the kernel driver <NUM> at the storage node <NUM> and is passed on for processing by various modules of the I/O stack <NUM> of the storage node <NUM>.

The I/O stack <NUM> can reside within the user address space <NUM> associated with a user process running on the node <NUM>. The I/O stack can include a frontend (FE) module <NUM>, a cache module <NUM>, a replication module <NUM>, and an RDMA module <NUM>. The FE module <NUM> is configured to associate the I/O associated with MA event <NUM> with a LUN object and validate the range (e.g., offset and length) of the operation associated with the MA event (e.g., READ or WRITE operation).

The cache module <NUM> is configured to perform caching of data and, in some aspects, satisfying the operation out of the cache. For example, the cache module <NUM> might first check if caching is available, and then it can fulfill a read cache function by checking for the presence of the read data in the cache. If the data is available, the read operation is satisfied out of the cache. Otherwise, the cache module <NUM> communicates the I/O to the next module in the stack.

The replication module <NUM> is, in a distributed storage environment, responsible for replicating data (e.g., on WRITE operation) to remote nodes for redundancy (the redundancy is provided by the remote node instead of being stored redundantly at node <NUM>).

The RDMA module <NUM> is configured to manage the RDMA connection to the peer nodes and manage communication operations between the local and remote nodes. For example, the RDMA module <NUM> can manage RDMA connection to storage node <NUM> via the communication link <NUM> to RDMA module <NUM>.

I/O processing within storage node <NUM> is similar to I/O processing described above in connection with storage node <NUM>. The kernel driver <NUM> can pass down a received memory access event for processing by various modules of the I/O stack <NUM> of the storage node <NUM>. The I/O stack <NUM> resides within the user address space <NUM> associated with a user process running on the node <NUM>. The I/O stack can include an FE module <NUM>, a cache module <NUM>, a replication module <NUM>, and an RDMA module <NUM>. The FE module <NUM>, the cache module <NUM>, the replication module <NUM>, and the RDMA module <NUM> performs similar functions as the corresponding modules of the I/O stack <NUM> discussed above.

Even though aspects described herein discuss allocation of the shared memory by a kernel driver, the disclosure is not limited in this regard and another OS module (e.g., a shared memory management module such as <NUM> or <NUM>) can be used to allocate and deallocate the shared memory (e.g., upon successful completion of a READ or WRITE operation associated with an MA event received from a client device).

<FIG> is a flowchart of a method suitable for RDMA memory access by a distributed storage node, if according to some example embodiments. The method <NUM> includes operations <NUM>, <NUM>, <NUM>, and <NUM>. By way of example and not limitation, the method <NUM> is described as being performed by the distributed storage node <NUM>. At operation <NUM>, a request for an input/output (I/O) process associated with data is received. For example, the distributed storage node <NUM> receives an MA event <NUM>, which includes a request for an I/O process (e.g., a READ or WRITE operation). At operation <NUM>, in response to the request, a memory segment shared between the operating system and a user process running on the distributed storage node is allocated using an operating system driver of the distributed storage node. For example, upon receiving the MA event <NUM>, a kernel driver (such as <NUM> or <NUM>) allocates memory <NUM> which is shared between the address space of the kernel <NUM> and a user process running within the user space <NUM>. The user process includes an I/O stack for processing the request, such as the I/O stack <NUM> illustrated in <FIG>. The shared memory segment includes a context memory portion (e.g., <NUM>) storing context information associated with the I/O stack, a header memory portion (e.g., <NUM>) storing header information for the I/O process, and a data memory portion (e.g., <NUM>) for storing the data. Additionally, the shared memory segment is registered for RDMA access with a target storage node (e.g., as part of RDMA registration of memory at system initialization). In this regard, shared memory <NUM> is registered for RDMA access on the distributed storage node <NUM> as well as the storage node <NUM>.

At operation <NUM>, an RDMA transfer is performed between the shared memory segment of the distributed storage node and the target storage node to complete the I/O process. For example, during a WRITE operation, RDMA transfer of header information and data is performed from storage node <NUM> to storage node <NUM>. After the WRITE operation is completed, status information is transferred via RDMA from storage node <NUM> to storage node <NUM>, as illustrated in <FIG>. During a READ operation, RDMA transfer of header information is performed from storage node <NUM> to storage node <NUM>, and RDMA transfer of data and status information is performed from storage node <NUM> to storage node <NUM> as illustrated in <FIG>. At operation <NUM>, the shared memory segment is deallocated in response to receiving a status indicator of completion of the RDMA transfer.

<FIG> is a block diagram illustrating a representative software architecture <NUM>, which may be used in conjunction with various device hardware described herein, according to some example embodiments. <FIG> is merely a non-limiting example of a software architecture <NUM> and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture <NUM> may be executing on hardware such as device <NUM> of <FIG> that includes, among other things, processor <NUM>, memory <NUM>, storage <NUM> and <NUM>, and I/O components <NUM> and <NUM>. A representative hardware layer <NUM> is illustrated and can represent, for example, the device <NUM> of <FIG>. The representative hardware layer <NUM> comprises one or more processing units <NUM> having associated executable instructions <NUM>. Executable instructions <NUM> represent the executable instructions of the software architecture <NUM>, including implementation of the methods, modules and so forth of <FIG>. Hardware layer <NUM> also includes memory and/or storage modules <NUM>, which also have executable instructions <NUM>. Hardware layer <NUM> may also comprise other hardware <NUM>, which represents any other hardware of the hardware layer <NUM>, such as the other hardware illustrated as part of device <NUM>.

In the example architecture of <FIG>, the software architecture <NUM> may be conceptualized as a stack of layers where each layer provides particular functionality. For example, the software architecture <NUM> may include layers such as an operating system <NUM>, libraries <NUM>, frameworks/middleware <NUM>, applications <NUM> and presentation layer <NUM>. Operationally, the applications <NUM> and/or other components within the layers may invoke application programming interface (API) calls <NUM> through the software stack and receive a response, returned values, and so forth illustrated as messages <NUM> in response to the API calls <NUM>. The layers illustrated in <FIG> are representative in nature and not all software architectures <NUM> have all layers. For example, some mobile or special purpose operating systems may not provide a frameworks/middleware <NUM>, while others may provide such a layer. Other software architectures may include additional or different layers.

The operating system <NUM> may manage hardware resources and provide common services. The operating system <NUM> may include, for example, a kernel <NUM>, services <NUM>, drivers <NUM>, and a shared memory management module <NUM>. The kernel <NUM> may act as an abstraction layer between the hardware and the other software layers. For example, kernel <NUM> may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and so on. The drivers <NUM> may be responsible for controlling or interfacing with the underlying hardware. For instance, the drivers <NUM> may include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth, depending on the hardware configuration.

In some aspects, the shared memory management module <NUM> may comprise suitable circuitry, logic, interfaces and/or code and can be configured to perform one or more of the functions discussed in connection with operations <NUM> - <NUM> in <FIG>, <NUM> - <NUM> in <FIG>, <NUM> - <NUM> in <FIG>, <NUM> - <NUM> in <FIG>, <NUM> - <NUM> in <FIG> <NUM> - <NUM> in <FIG>, and <NUM> - <NUM> in <FIG>. Additionally, the shared memory management module <NUM> performs functionalities in connection with allocating shared memory used for performing I/O processing and deallocating the shared memory after completion of the I/O processing. In some aspects, functionalities of the shared memory management module <NUM> can be performed by other operating system modules, such as <NUM>, <NUM>, or <NUM>.

The libraries <NUM> may provide a common infrastructure that may be utilized by the applications <NUM> and/or other components and/or layers. The libraries <NUM> typically provide functionality that allows other software modules to perform tasks in an easier fashion than to interface directly with the underlying operating system <NUM> functionality (e.g., kernel <NUM>, services <NUM>, drivers <NUM>, and/or module <NUM>). The libraries <NUM> may include system libraries <NUM> (e.g., C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, the libraries <NUM> may include API libraries <NUM> such as media libraries (e.g., libraries to support presentation and manipulation of various media format such as MPEG4, H. <NUM>, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., an OpenGL framework that may be used to render 2D and 3D in a graphic content on a display), database libraries (e.g., SQLite that may provide various relational database functions), web libraries (e.g., WebKit that may provide web browsing functionality), and the like. The libraries <NUM> may also include a wide variety of other libraries <NUM> to provide many other APIs to the applications <NUM> and other software components/modules.

The frameworks/middleware <NUM> (also sometimes referred to as middleware) may provide a higher-level common infrastructure that may be utilized by the applications <NUM> and/or other software components/modules. For example, the frameworks/middleware <NUM> may provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks/middleware <NUM> may provide a broad spectrum of other APIs that may be utilized by the applications <NUM> and/or other software components/modules, some of which may be specific to a particular operating system <NUM> or platform.

The applications <NUM> include built-in applications <NUM> and/or third-party applications <NUM>. Examples of representative built-in applications <NUM> may include but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and/or a game application. Third-party applications <NUM> may include any of the built-in applications <NUM> as well as a broad assortment of other applications. In a specific example, the third-party application <NUM> (e.g., an application developed using the Android™ or iOS™ software development kit (SDK) by an entity other than the vendor of the particular platform) may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or other mobile operating systems. In this example, the third-party application <NUM> may invoke the API calls <NUM> provided by the mobile operating system such as operating system <NUM> to facilitate functionality described herein.

The applications <NUM> may utilize built-in operating system functions (e.g., kernel <NUM>, services <NUM>, drivers <NUM>, and/or module <NUM>), libraries (e.g., system libraries <NUM>, API libraries <NUM>, and other libraries <NUM>), and frameworks/middleware <NUM> to create user interfaces to interact with users of the system. Alternatively, or additionally, in some systems, interactions with a user may occur through a presentation layer, such as presentation layer <NUM>. In these systems, the application/module "logic" can be separated from the aspects of the application/module that interact with a user.

Some software architectures utilize virtual machines. In the example of <FIG>, this is illustrated by virtual machine <NUM>. A virtual machine creates a software environment where applications/modules can execute as if they were executing on a hardware machine (such as the device <NUM> of <FIG>, for example). A virtual machine <NUM> is hosted by a host operating system (operating system <NUM> in <FIG>) and typically, although not always, has a virtual machine monitor <NUM>, which manages the operation of the virtual machine <NUM> as well as the interface with the host operating system (i.e., operating system <NUM>). A software architecture <NUM> executes within the virtual machine <NUM> such as an operating system <NUM>, libraries <NUM>, frameworks/middleware <NUM>, applications <NUM>, and/or presentation layer <NUM>. These layers of software architecture executing within the virtual machine <NUM> can be the same as corresponding layers previously described or may be different.

<FIG> is a block diagram illustrating circuitry for a device that implements algorithms and performs methods, according to some example embodiments. All components need not be used in various embodiments. For example, clients, servers, and cloud-based network devices may each use a different set of components, or in the case of servers, for example, larger storage devices.

One example computing device in the form of a computer <NUM> (also referred to as computing device <NUM>, computer system <NUM>, or computer <NUM>) may include a processor <NUM>, memory storage <NUM>, removable storage <NUM>, non-removable storage <NUM>, input interface <NUM>, output interface <NUM>, and communication interface <NUM>, all connected by a bus <NUM>. Although the example computing device is illustrated and described as the computer <NUM>, the computing device may be in different forms in different embodiments.

The memory storage <NUM> may include volatile memory <NUM> and non-volatile memory <NUM> and may store a program <NUM>. The computer <NUM> may include - or have access to a computing environment that includes - a variety of computer-readable media, such as the volatile memory <NUM>, the non-volatile memory <NUM>, the removable storage <NUM>, and the non-removable storage <NUM>. Computer storage includes random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

Computer-readable instructions stored on a computer-readable medium (e.g., the program <NUM> stored in the memory <NUM>) are executable by the processor <NUM> of the computer <NUM>. A hard drive, CD-ROM, and RAM are some examples of articles including a non-transitory computer-readable medium such as a storage device. The terms "computer-readable medium" and "storage device" do not include carrier waves to the extent that carrier waves are deemed too transitory. "Computer-readable non-transitory media" includes all types of computer-readable media, including magnetic storage media, optical storage media, flash media, and solid-state storage media. It should be understood that software can be installed in and sold with a computer. Alternatively, the software can be obtained and loaded into the computer, including obtaining the software through a physical medium or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example. As used herein, the terms "computer-readable medium" and "machine-readable medium" are interchangeable.

The program <NUM> may utilize a customer preference structure using modules discussed herein, such as a shared memory management module <NUM>. The shared memory management module <NUM> may be the same as the shared memory management module <NUM> as discussed in connection with <FIG>.

Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine, an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any suitable combination thereof). Moreover, any two or more of these modules may be combined into a single module, and the functions described herein for a single module may be subdivided among multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

In some aspects, module <NUM>, as well as one or more other modules that are part of the program <NUM>, can be integrated as a single module, performing the corresponding functions of the integrated modules.

Although a few embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Other embodiments may be within the scope of the following claims.

It should be further understood that software including one or more computer-executable instructions that facilitate processing and operations as described above with reference to any one or all of steps of the disclosure can be installed in and sold with one or more computing devices consistent with the disclosure. Alternatively, the software can be obtained and loaded into one or more computing devices, including obtaining the software through physical medium or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example.

Also, it will be understood by one skilled in the art that this disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the description or illustrated in the drawings. The embodiments herein are capable of other embodiments and capable of being practiced or carried out in various ways. Also, it will be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. Unless limited otherwise, the terms "connected," "coupled," and "mounted," and variations thereof herein are used broadly and encompass direct and indirect connections, couplings, and mountings. In addition, the terms "connected" and "coupled", and variations thereof, are not restricted to physical or mechanical connections or couplings. Further, terms such as up, down, bottom, and top are relative, and are employed to aid illustration, but are not limiting.

The components of the illustrative devices, systems, and methods employed in accordance with the illustrated embodiments can be implemented, at least in part, in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. These components can be implemented, for example, as a computer program product such as a computer program, program code or computer instructions tangibly embodied in an information carrier, or in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus such as a programmable processor, a computer, or multiple computers.

A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other units suitable for use in a computing environment. Also, functional programs, codes, and code segments for accomplishing the techniques described herein can be easily construed as within the scope of the claims by programmers skilled in the art to which the techniques described herein pertain. Method steps associated with the illustrative embodiments can be performed by one or more programmable processors executing a computer program, code or instructions to perform functions (e.g., by operating on input data and/or generating an output). Method steps can also be performed by, and apparatus for performing the methods can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit), for example.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an ASIC, a FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.

Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The required elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example, semiconductor memory devices, e.g., electrically programmable read-only memory or ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, and data storage disks (e.g., magnetic disks, internal hard disks, or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks). The processor and the memory can be supplemented by or incorporated in special purpose logic circuitry.

Those of skill in the art understand that information and signals may be represented using any of a variety of different technologies and techniques.

As used herein, "machine-readable medium" (or "computer-readable medium") means a device able to store instructions and data temporarily or permanently and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., Erasable Programmable Read-Only Memory (EEPROM)), and/or any suitable combination thereof. The term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store processor instructions. The term "machine-readable medium" shall also be taken to include any medium, or a combination of multiple media, that is capable of storing instructions for execution by one or more processors <NUM>, such that the instructions, when executed by one or more processors <NUM>, cause the one or more processors <NUM> to perform any one or more of the methodologies described herein. Accordingly, a "machine-readable medium" refers to a single storage apparatus or device, as well as "cloud-based" storage systems or storage networks that include multiple storage apparatus or devices. The term "machine-readable medium" as used herein excludes signals per se.

In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the scope disclosed herein.

Although the present disclosure has been described with reference to specific features and embodiments thereof, it is evident that various modifications and combinations can be made thereto without departing from the scope of the disclosure. For example, other components may be added to, or removed from, the described systems. The specification and drawings are, accordingly, to be regarded simply as an illustration of the disclosure as defined by the appended claims.

Claim 1:
A computer-implemented method for remote direct memory access, RDMA, by a distributed storage node (<NUM>), the method comprising:
receiving a request for an input/output, I/O, process associated with data, wherein the I/O process comprises a READ or WRITE operation;
in response to the request, allocating using an operating system driver of the distributed storage node (<NUM>), a memory segment (<NUM>) shared between the operating system and a user process running on the distributed storage node (<NUM>), wherein:
the user process includes an I/O stack for processing the request;
the shared memory segment (<NUM>) includes a context memory portion (<NUM>) storing context information associated with the I/O stack, a header memory portion (<NUM>) storing header information for the I/O process, and a data memory portion (<NUM>) for storing the data; and
the shared memory segment (<NUM>) is registered for RDMA access with a target storage node (<NUM>) comprising a target RDMA memory (<NUM>);
the target RDMA memory (<NUM>) includes a header portion (<NUM>) and a data portion (<NUM>);
performing an RDMA transfer between the shared memory segment (<NUM>) of the distributed storage node (<NUM>) and the target storage node (<NUM>) to complete the I/O process, wherein during a WRITE operation header information associated with the WRITE operation and stored in the header memory portion (<NUM>) of the shared memory segment (<NUM>) as well as data stored in the data memory portion (<NUM>) of the shared memory segment (<NUM>) is transferred to the header portion (<NUM>) and the data portion (<NUM>) respectively of the target RDMA memory (<NUM>); and
wherein during a READ operation header information associated with the READ operation and stored in the header memory portion (<NUM>) of the shared memory segment (<NUM>) is transferred to the header portion (<NUM>) of the target RDMA memory (<NUM>) and data stored in the data portion (<NUM>) of the target RDMA memory (<NUM>) is transferred to the data memory portion (<NUM>) of the shared memory segment (<NUM>); and
deallocating the shared memory segment (<NUM>) in response to receiving a status indicator of completion of the RDMA transfer.