Method for aggregated NVME-over-fabrics ESSD

In a method of storage aggregation for NVMe Over Fabrics devices, the method includes: identifying an aggregation group as an aggregated Ethernet SSD comprising a plurality of NVMe-oF SSDs; selecting one of the NVMe-oF SSDs of the aggregation group as a primary NVMe-oF SSD; selecting others of the NVMe-oF SSDs of the aggregation group as secondary NVMe-oF SSDs; and initializing a Map Allocation Table in the primary NVMe-oF SSD with a processor for managing the NVMe-oF SSDs.

BACKGROUND

Some embodiments of the present disclosure relate generally to a system and method for aggregating multiple memory drives (e.g., eSSDs) to be perceived by a host as a single, large, logical capacity.

2. Description of the Related Art

Solid State Drives (SSDs) are rapidly becoming preferred storage elements of modern IT infrastructures, thereby replacing traditional Hard Disk Drives (HDDs). SSDs offer very low latency, high data read/write throughput, and reliable data storage.

NVMe-over-Fabrics (NVMe-oF) is an emerging technology that allows hundreds or thousands of NVMe-oF devices (e.g., non-volatile memory (NVMe) SSDs) to be connected over network fabrics such as IB, FC, and Ethernet. The NVMe-oF protocol enables remote Direct Attach Storage (rDAS) implementation. This allows a large number of SSDs to be connected to a remote host. The NVMe-oF protocol uses the Remote Direct Memory Access (RDMA) protocol to provide reliable communication of NVMe commands, data, and responses. Transport protocols for providing RDMA services include iWARP, RoCE v1, and RoCE v2.

The NVMe-oF interface allows a large number of SSDs to be connected to a remote host. Conventionally, for each NVMe-oF SSD, a driver instance runs on the remote host. For some applications, the storage capacity provided by a single SSD may be insufficient

SUMMARY

Some embodiments of the present disclosure provide a method of aggregating a plurality of SSDs that are perceived by a host as a single large-capacity logical volume, and a networked structure for accomplishing the method.

According to some embodiments, in a method of storage aggregation for NVMe Over Fabrics devices, the method includes: identifying an aggregation group as an aggregated Ethernet SSD comprising a plurality of NVMe-oF SSDs; selecting one of the NVMe-oF SSDs of the aggregation group as a primary NVMe-oF SSD; selecting others of the NVMe-oF SSDs of the aggregation group as secondary NVMe-oF SSDs; and initializing a Map Allocation Table in the primary NVMe-oF SSD with a processor for managing the NVMe-oF SSDs.

According to some example embodiments, initializing the Map Allocation Table with the processor occurs under guidance of a storage administrator connected to the aggregation group.

According to some example embodiments, the Map Allocation Table comprises, for each of the NVMe-oF SSDs of the aggregation group, a capacity of the NVMe-oF SSD, an address of the NVMe-oF SSD, and a remaining amount of capacity of the NVMe-oF SSD.

According to some example embodiments, the method further includes providing the address of the primary NVMe-oF SSD to user applications to enable data transfer between the aggregation group and the user applications.

According to some example embodiments, the method further includes: receiving an Admin command at the primary NVMe-oF SSD from a host connected to the aggregation group; determining whether data corresponding to the Admin command is stored on only the primary NVMe-oF SSD or on one or more of the secondary NVMe-oF SSDs; when the data is stored on the one or more of the secondary NVMe-oF SSDs, segmenting the Admin command into one or more Admin sub-commands respectively corresponding to the one or more of the secondary NVMe-oF SSDs; transferring the data to the host; receiving sub-command completion entries from the one or more of the secondary NVMe-oF SSDs; and creating and sending a completion entry from the primary NVMe-oF SSD to the host.

According to some example embodiments, the method further includes: receiving an Admin sub-command of the one or more Admin sub-commands at a corresponding secondary NVMe-oF SSD of the one or more of the secondary NVMe-oF SSDs; determining whether to transfer the data from the corresponding secondary NVMe-oF SSD to the primary NVMe-oF SSD in accordance with the Admin sub-command; creating a completion entry; and sending the completion entry to the primary NVMe-oF SSD.

According to some example embodiments, the method further includes: receiving a command to create a Namespace or a command to delete a Namespace at the primary NVMe-oF SSD; referencing the Map Allocation Table with the primary NVMe-oF SSD; allocating capacity in the primary NVMe-oF SSD and/or in one or more of the secondary NVMe-oF SSDs when the command is to create the Namespace, or retrieving a corresponding one of the primary NVMe-oF SSD and/or the one or more of the secondary NVMe-oF SSDs wen the command is to delete the Namespace; and updating the Map Allocation Table.

According to some example embodiments, the method further includes: receiving a Read/Write command at the primary NVMe-oF SSD; looking up the Map Allocation Table with the primary NVMe-oF SSD; creating one or more Read/Write sub-commands; sending the one or more Read/Write sub-commands to one or more of the secondary NVMe-oF SSDs, respectively; transferring data between a host and the primary NVMe-oF SSD and/or the one or more of the secondary NVMe-oF SSDs in accordance with the Read/Write command; and sending a completion to the host following transfer of the data.

According to some example embodiments, the method further includes: receiving a Read/Write sub-command of the one or more Read/Write sub-commands at a corresponding secondary NVMe-oF SSD of the one or more secondary NVMe-oF SSDs; extracting transport information corresponding to the Read/Write sub-command; issuing a Read/Write request from the corresponding secondary NVMe-oF SSD to the host; and sending a completion entry to the primary NVMe-oF SSD following completion of a data transfer corresponding to the Read/Write sub-command.

According to some embodiments, in a method of NVMe-oF SSD capacity aggregation in a group of NVMe-oF Ethernet SSDs, the method includes: identifying a plurality of NVMe-oF SSDs of an aggregation group; assigning one of the NVMe-oF SSDs as a primary NVMe-oF SSD; and assigning remaining ones of the NVMe-oF SSDs as secondary NVMe-oF SSDs, wherein the only NVMe-oF SSD that is visible to a host driver of a host is the primary NVMe-oF SSD.

According to some example embodiments, the method further includes maintaining a Map Allocation Table with the primary NVMe-oF SSD in accordance with commands received by the primary NVMe-oF SSD from the host, wherein the Map Allocation Table indicates a logical block address (LBA) space divided among the primary NVMe-oF SSD and one or more of the secondary NVMe-oF SSDs of the aggregation group.

According to some example embodiments, the method further includes initializing a Map Allocation Table with a processor to configure the aggregation group in accordance with assigning one of the NVMe-oF SSDs as the primary NVMe-oF SSD.

According to some example embodiments, the method further includes aggregating capacities of the secondary NVMe-oF SSDs with the primary NVMe-oF SSD in accordance with a command received by the primary NVMe-oF SSD from the host such that the plurality of NVMe-oF SSDs of the aggregation group appear to the host as a single, aggregated logical capacity.

According to some example embodiments, the method further includes: using the primary NVMe-oF SSD to allocate capacity to one or more of the secondary NVMe-oF SSDs; and using the primary NVMe-oF SSD to record the allocated capacity and associated mapped logical block address (LBA) ranges in a Map Allocation Table.

According to some example embodiments, the method further includes overprovisioning aggregated capacity of the secondary NVMe-oF SSDs and the primary NVMe-oF SSD with the primary NVMe-oF SSD.

According to some example embodiments, the method further includes: receiving a command from the host at the primary NVMe-oF SSD; dividing the command into a plurality of sub-commands each corresponding to a respective one of corresponding ones of the secondary NVMe-oF SSDs; and sending the sub-commands from the primary NVMe-oF SSD to the corresponding ones of the secondary NVMe-oF SSDs.

According to some example embodiments, the method further includes directly transferring data from the corresponding ones of the secondary NVMe-oF SSDs to the host based on the respective sub-commands.

According to some example embodiments, the method further includes: receiving the respective sub-commands at the corresponding ones of the secondary NVMe-oF SSDs from the primary NVMe-oF SSD; performing a task corresponding to the respective sub-commands; and sending respective sub-command completion entries from the corresponding ones of the secondary NVMe-oF SSDs to the primary NVMe-oF SSD upon completion of the task.

According to some example embodiments, the method further includes: using the primary NVMe-oF SSD to maintain a sub-command context table; receiving the sub-command completion entries at the primary NVMe-oF SSD from the secondary NVMe-oF SSDs; and tracking execution of the sub-commands with the primary NVMe-oF SSD in accordance with the received sub-command completion entries.

According to some embodiments, there is provided an aggregated Ethernet SSD group including: an Ethernet SSD chassis; an Ethernet switch on the Ethernet SSD chassis for enabling communication with a host driver; a processor coupled to the Ethernet switch; a PCIe switch coupled to the board management controller; a plurality of NVMe-oF SSDs including: a primary NVMe-oF SSD; and a plurality of secondary NVMe-oF SSDs connected to the primary NVMe-oF SSD via a private communication channel comprising the Ethernet switch and the PCIe switch, wherein only the primary NVMe-oF SSD is visible to the host, and wherein the board management controller is configured to initially determine which of the NVMe-oF SSDs comprises the primary NVMe-oF SSD.

Accordingly, because a single primary eSSD performs all of the NVMe-oF protocol processing while tracking completion of all relevant sub-commands by the secondary eSSDs and keeping the secondary eSSDs invisible to the host, the aggregation group of eSSDs is presented as a single, large logical capacity to the host.

DETAILED DESCRIPTION

It will be understood that when an element, layer, region, or component is referred to as being “on,” “connected to,” or “coupled to” another element, layer, region, or component, it can be directly on, connected to, or coupled to the other element, layer, region, or component, or one or more intervening elements, layers, regions, or components may be present. In addition, it will also be understood that when an element or layer is referred to as being “between” two elements or layers, it can be the only element or layer between the two elements or layers, or one or more intervening elements or layers may also be present.

For the purposes of this disclosure, “at least one of X, Y, and Z” and “at least one selected from the group consisting of X, Y, and Z” may be construed as X only, Y only, Z only, or any combination of two or more of X, Y, and Z, such as, for instance, XYZ, XYY, YZ, and ZZ. Like numbers refer to like elements throughout. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

FIG. 1depicts a block diagram of a system architecture100used in NVMe-oF Ethernet SSD (eSSD) storage including multiple aggregated eSSDs110in a single eSSD chassis120, according to an embodiment of the present disclosure.FIG. 2depicts a block diagram of multiples of the eSSD chassis120such as those shown in the embodiment ofFIG. 1connected together in an Ethernet SSD rack, according to an embodiment of the present disclosure.

As described above, the NVMe-oF interface allows a large number of SSDs110to be connected to a remote host190. Conventionally, for each NVMe-oF SSD110, a driver instance runs on the remote host190. However, for some applications, the storage capacity provided by a single SSD110is insufficient. Such applications may benefit from a single logical volume having a capacity of hundreds of terabytes. Accordingly, such applications may benefit from embodiments of the present disclosure, which provide a large number of individual SSDs110that are aggregated together in an “Aggregation Group,” and that are presented as a single logical volume to the application.

For example, 24 16-terabyte (16 TB) eSSDs can be presented as a single, logical 384 TB drive. Some examples of applications needing a large number of aggregated SSDs110include big data-mining and analytics, petro-chemicals, gas and energy explorations, experimental particle physics, and pharmaceutical drug development. These examples may require High Performance Computing (HPC), which needs both large storage capacities and high performance.

Although it is possible to have a system software layer that aggregates underlying SSDs110, and that provides a single logical, expansive volume, such system software is generally highly complex and sophisticated. Such software may require a large number of NVMe-oF driver instances running on the host190, thereby consuming system resources such as memory, CPU cycles, and power. Target side solutions could potentially use either x86 servers or RAID-on-Chip (ROC) systems to provide a large capacity as single logical volume. However, such solutions are generally complex, expensive, and have performance and energy penalties. For example, receiving and transmitting data by using a CPU may consume an amount of energy that is several times larger than energy consumed by a DMA engine, an ASIC, etc. according to embodiments of the present invention.

Accordingly, embodiments of the present disclosure provide a method and structure for use in Ethernet NVMe-oF SSDs for aggregating multiple eSSDs110in an efficient and cost-effective manner.

Referring toFIG. 1, the eSSDs110are assigned one of two roles (e.g., at the direction of a storage administrator), such that each eSSD110acts as either a primary eSSD (P-eSSD)110por a secondary eSSD (S-eSSD)110s. A single P-eSSD110pand a set of multiple S-eSSDs110sin a single chassis120(or in multiple chassis120in a given rack, or in multiple chassis120in multiple racks230that are distributed over a wide area) collectively provide the requisite flash memory capacity used by a remote host190as a single logical drive. An eSSD chassis120includes the eSSDs110, along with a processor, such as board management controller (BMC) device150, and an Ethernet switch160for external connectivity. Although the eSSD chassis120is used to refer to a group of NVMe-oF devices in the description of the embodiments below, other embodiments of the present invention can be similarly applied to any other plurality of NVMe-oF devices, regardless of a physical housing thereof (e.g., a chassis, a rack, or a container-based housing). Further, although eSSDs110are used to describe the NVMe-oF devices of the embodiments described below, other NVMe-oF devices may equally applied to other embodiments of the present invention.

Accordingly, via corresponding Ethernet switches160, it is possible for the aggregation of eSSDs110to span across multiple chassis120in a rack230, as well as to span across multiple racks230each including multiple chassis120.

The P-eSSD110pis the only eSSD110that is visible to the remote host NVMe-oF driver170, and therefore terminates the NVMe-oF protocol. The P-eSSD110ppresents a single, large, aggregated logical capacity to the host190on behalf of itself and all of the remaining S-eSSDs110sin a same aggregation group. The P-eSSD110preceives all of the input/output (I/O) commands180from the remote host NVMe-oF driver170, and provides command responses (e.g., completion entries)182to the host190.

The P-eSSD110palso maintains a Map Allocation Table (MAT) that indicates the logical block address (LBA) space divided among the P-eSSD110palong with some or all of the S-eSSDs110sof the same aggregation group of eSSDs110. When an I/O command180is received by the P-eSSD110p, the P-eSSD110pfirst looks up the MAT (e.g., MAT300ofFIG. 3, described further below) to determine which of the eSSDs110(e.g., the P-eSSD110p, one or more of the S-eSSDs110s, or a set of both) can satisfy the I/O command180. In accordance with the MAT, the P-eSSD110pthen sends appropriately modified NVMe-oF I/O sub-commands132to the set of appropriate S-eSSDs110s.

To send the sub-commands132, the P-eSSD110palso establishes a private Ethernet-RDMA connection (or a proprietary communication channel)130over the PCIe bus140and the control plane135with each of the S-eSSDs110safter powering on. This private queue-pair (QP) communication channel130is used by the P-eSSD110pto send I/O commands (e.g., sub-commands132) to the S-eSSDs110s, and to receive completion entries134from the S-eSSDs110s. The private communication channel130can be Ethernet, and can enable data to be transmitted over the Ethernet switch160. However, the private communication channel130can also be a PCIe-based channel, and can enable data to be transmitted over a PCIe switch. That is, it is possible that all of the eSSDs110may use two or more modes of communication to communicate with each other. For example, an Ethernet channel may generally be used for transmission of data, and a PCIe channel may be used for management, while either channel can be used as the private communication channel130.

The S-eSSDs110sare normal NVMe-oF SSDs110that perform only data transfers to and from the remote host190using NVMe-oF protocol. These data transfers are done directly with the remote host190using RDMA READ and WRITE services. The S-eSSDs110sreceive commands (e.g., sub-commands132) from the P-eSSD110p, but do not receive commands directly from the remote host NVMe-oF driver170. The S-eSSDs110ssend sub-command completion entries134to the P-eSSD110p, instead of to the remote host190, to indicate completion of the sub-commands132.

The P-eSSD110phandles all of the NVMe-oF protocol termination, handles all of the host command and completion queuing (e.g., Submission Queues/Completion Queues (SQ/CQ)), and is visible to the remote host NVMe-oF driver170running on the remote host initiator. When the remote host driver170issues an NVMe Admin command180or an I/O command180, the command180is issued to the P-eSSD110p, and all of the Admin commands180are executed by the P-eSSD110p. The I/O commands180, however, can be spread between multiple eSSDs110.

The P-eSSD110pmay also perform its own share of data transfers in accordance with the I/O command180. The P-eSSD110pthen waits for all of the sub-command completion entries134to arrive (e.g., from the set of S-eSSDs110s) on the private communications channel130before sending a command completion entry182, which corresponds to the original command180, to the remote host190.

The P-eSSD110palso maintains a “command context” for each command in execution in a command context table (e.g., seeFIG. 6). This command context is used by the P-eSSD110pto track the execution of the sub-commands132, data transfers, and any error status. When all of the sub-commands132are done, a command response/completion entry182is provided to the remote host190, and the command context table is deallocated.

Referring toFIG. 2, multiple eSSD chassis120may be connected together in an Ethernet SSD rack230, wherein a Top-Of-Rack (TOR) switch240is used to provide connectivity between the multiple chassis120in a common rack230. Similarly, multiple racks230that are located at different respective geographical locations can be connected to each other through respective TOR switches240, which are either directly connected to each other, or connected to each other through external switches. The Ethernet racks230can be within a single datacenter building, or can be distributed across a wide geographical area.

To summarize, embodiments of the present disclosure provide a mechanism for aggregating multiple Ethernet NVMe-oF SSDs (eSSDs)110to be presented as a single, large capacity NVMe-oF SSD. The eSSDs110can be located in a single chassis120, can be located in multiple chassis120in a single rack230, or can even be scattered across a large number of Ethernet racks230each having multiple chassis120. One of the eSSDs110is assigned the role of a primary eSSD (P-eSSD)110p. The other eSSDs110are assigned the role of a secondary eSSD110s(S-eSSD). The S-eSSDs110sreceive sub-commands132from the P-eSSD110p, complete the sub-commands132, and send completion entries134for those sub-commands132back to the P-eSSD110p, although the S-eSSDs110sperform direct data transfers with the remote host initiator. Accordingly, the present embodiment allows aggregation of capacities to effectively act as a single Ethernet SSD without sacrificing any storage bandwidth.

FIG. 3is an example of a “Map Allocation Table”300maintained by a P-eSSD110p, according to an embodiment of the present disclosure.FIG. 4depicts a flowchart400depicting initialization of the Map Allocation Table300, according to an embodiment of the present embodiment.

Referring toFIGS. 3 and 4, as described above, the present embodiment utilizes two types of eSSDs110(e.g., the P-eSSD110pand the S-eSSDs110s). Both P-eSSD110pand S-eSSD110suse NVMe over Fabrics protocol to provide storage service to the host190. The P-eSSD110pmaintains a table (e.g., the Map Allocation Table (MAT)300) containing details of the S-eSSDs110sin the Aggregation Group of eSSDs110, which are perceived by the host190as a single logical volume.

The MAT300may be initialized by the BMC150in the same chassis120as the P-eSSD110p. The BMC150may manage the Ethernet chassis120and the components like the Ethernet switch160and the eSSDs110. The BMC150has PCIe and SMBus interfaces for system management purposes. Also, the BMC will determine which eSSDs110will be aggregated (S410) (e.g., under the direction of the storage administrator), and when the eSSDs110are determined, the BMC150may configure the Ethernet switch160.

The three columns on the left side of the MAT300are initialized by the BMC150under guidance of a storage administrator. The BMC150and the storage administrator are visible to, and have knowledge of, all of the eSSDs110present in the Aggregation Group/storage system. Such knowledge includes capacities311and address locations312of each of the eSSDs110. The storage administrator may decide which S-eSSDs110sare needed to form an “Aggregated Ethernet SSD” (e.g., the Aggregation Group). The BMC150and the storage administrator may advertise, or provide, the network address of the P-eSSD110pto the users so that a user application corresponding to the remote host NVMe-oF driver170knows where to find the Aggregated Ethernet SSD. The BMC150and the storage administrator may also select or nominate one of the eSSDs110as the P-eSSD110p(S420), and may also change the eSSD110that is designated as the P-eSSDs110pfor one or more of a variety of reasons after the initial designation. Thereafter, the BMC150may program primary and secondary modes of the Aggregation Group (S430).

The P-eSSD110pmay also keep a copy of the MAT300on the BMC150, and may regularly update the copy of the MAT300stored with the BMC150. In some embodiments, only the P-eSSD110pcontains the official MAT300, and an eSSD index313of “0” denotes the P-eSSD110p, with the remaining eSSD index values corresponding to respective ones of the S-eSSDs110s.

The P-eSSD110pterminates the NVMe-oF protocol for the host driver170, and executes all of the commands180issued by the host driver170. When the host190commands180are complete, the P-eSSD110psends the completion entries182in the form of “Completion Entries”182back to the host190driver170. With respect to the host commands180, the remote host NVMe-oF driver170is completely unaware of the presence of the S-eSSDs110s. The P-eSSD110palso maintains the Submission Queues (SQ), and submits the command completions to Completion Queues (CQ).

The three columns on the right of the MAT300are updated and maintained by the P-eSSD110p. When the remote host NVMe-oF driver170creates a “Namespace,” certain flash capacity is allocated to that Namespace. The Namespace LBA range314is mapped to a set of eSSDs110, and is recorded in the MAT300maintained by the P-eSSD110p. The details of this process will be described with respect toFIG. 8below.

The P-eSSD110pmay also perform certain initializations. Once the MAT300is initialized in the p-eSSD110p(S440), the P-eSSD110pis aware of which of the eSSDs110are the corresponding S-eSSDs110s. The P-eSSD110pthen sets up a communication channel130with each of the S-eSSDs110sin the Aggregation Group. The communication channel130can be over the Ethernet interface going through the Ethernet switch160in the chassis120, or alternatively can be over a PCIe interface passing through a PCIe switch in the chassis120. If one of the S-eSSDs110sis located in different chassis120that is located in the same rack230, the communication channel130is established through the TOR switch240. A communication channel130within a given chassis120can be over a PCIe bus140as well.

FIG. 5depicts a block diagram of a system architecture500used in NVMe-oF Ethernet SSD (eSSD) storage including data flows to and from multiple aggregated eSSDs110in multiple eSSD chassis120respectively located in multiple racks230, according to an embodiment of the present disclosure.

Referring toFIG. 5, it is possible for the P-eSSD110pto establish Ethernet communication channels530with S-eSSDs110slocated in a wide area network (WAN) through external network switches and routers. Such a private Ethernet communication channel530can be RDMA Queue-pair (QP), or can be a proprietary method. The Ethernet communication channel530is used to exchange sub-commands132and associated completions.

FIG. 6is a table600depicting an example command context.

Referring toFIG. 6, each of the sub-commands132have a command ID640, and carry a “Command Tag,”610so that when the P-eSSD110preceives the completion entries134, the completion entries134can be traced back to the original command180. Upon tracing back the completion entries134of the sub-commands132, a “# of Sub-commands” field620is decremented, and a received error status is latched with the current status. When the # of Subcommands field620reaches zero, the corresponding command180that is the parent of the sub-commands132is complete, and the P-eSSD110pmay send a completion182back to the remote host190. The P-eSSD110pat that point generates a completion entry with accumulated error statuses630, and puts it in the associated CQ.

FIG. 7depicts a flowchart700depicting handling of Admin commands180by the P-SSD110p, according to an embodiment of the present disclosure.

ReferringFIGS. 1 and 7, and as described above, each P-eSSD110pmaintains the command SQs. When there exists commands180that are received by the P-eSSD110P (S710) and that are available for execution, the P-eSSD110parbitrates the SQs, and then selects a command180for execution. The P-eSSD110pexecutes all of the NVMe commands (e.g., Admin commands and I/O commands)180. That is, although the S-eSSDs110smay send data directly to the host, the S-eSSDs110sdo not directly receive commands from the host190, and do not send completions directly to the host190. The Admin command execution by the P-eSSD110pmay not need any communication with any S-eSSD110s.

After receiving the command180, the P-eSSD110P determines where the data is, and whether it has access to all of the data (S720). If the P-eSSD110phas all of the data, the P-eSSD110ptransfers the data to the host190(S770).

If the P-eSSD110pdetermines that it does not have all of the data (S720), the P-eSSD110pthen consults the MAT300to determine where the requested data is located (S730). Once the set of relevant eSSDs110is identified, the P-eSSD110pproceeds with the execution of that command180. When all of the relevant requested data is available in the P-eSSD110pitself, the P-eSSD110pperforms the data transfer185. However, when the requested data is scattered over a set of P-eSSD110pand/or S-eSSDs110s, the P-eSSD110pdivides the original command180into an appropriate number of sub-commands132(S740). The number of sub-commands132corresponds to the number of the eSSDs110over which the requested data is scattered. Each sub-command132corresponds to the portion of the requested data that each eSSD110possesses.

The P-eSSD110pputs appropriate Start LBA (SLBA), Number of Blocks (NLB), and remote Scatter/Gather Lists (SGLs) in the sub-commands132. The SGL contains address, key, and size of the transfer buffer on the remote host190. The P-eSSD110pthen sends those sub-commands132to the respective S-eSSDs110s(S750) over the private QP communication channels130in a command segmentation process, and waits to receive completion entries134from each of the respective S-eSSDs110s(S760). Accordingly, the original command180is segmented into sub-commands132so that appropriate eSSDs110can perform data transfers in parallel, thereby enabling the data to be transferred to the host (S770).

The P-eSSD110pcreates a command context for the commands180in execution, which is described with respect toFIG. 6. The command context is used to keep track of the execution of the sub-commands132, as well as any intermediate error status of the sub-commands132(S780). Once the P-eSSD110pconfirms that the Admin command is complete, the P-eSSD110psends a completion entry to the host (S790).

Accordingly, all of the Admin commands180issued by the remote host NVMe-oF driver170are received (S710) and executed by P-eSSD110p. The P-eSSD110pmay possibly complete many or all of the Admin commands180alone (e.g., when the P-eSSD110pdetermines in (S720) that it, alone, has all of the information necessary to complete the Admin commands). In some cases the P-eSSD110pmay fetch certain pieces of information from the S-eSSDs110sbefore completing an Admin command180. If the P-eSSD110pseeks certain non-user data information from the S-eSSDs110s, the P-eSSD110pcan create and send Admin sub-commands132to the respective S-eSSDs110s. The S-eSSDs110scan send back any necessary data and sub-command completion entries134to the P-eSSD110pusing the private communications channel130between them.

FIG. 8depicts a flowchart800depicting the execution of Namespace Create and Delete commands, according to an embodiment of the present disclosure.

Referring toFIG. 8, the P-eSSD110pmay receive and execute Namespace Create commands (S810) and/or Delete Admin commands (S811). When the P-eSSD110preceives a Namespace Create command (S810), the P-eSSD110pmay look up the MAT300(S820), and may allocate an appropriate amount of capacity from the total available pool (S830). A newly created Namespace may have flash capacity solely from the P-eSSD110p, or solely from certain ones of the S-eSSD(s), or the newly created Namespace may have the flash capacity from any combination of P-eSSD110pand S-eSSDs110s. The P-eSSD110pmay then record the allocated capacity and associated mapped LBA ranges in the MAT300(S840).

When the P-eSSD110preceives a Namespace Delete command (S811) for deleting a Namespace, the P-eSSD110plooks up the MAT300(S821), retrieves the corresponding eSSDs (S831), and deallocates the associated capacity (S841), thereafter updating the MAT300accordingly.

With respect to Namespace Create/Delete command execution by the S-eSSDs110s, the S-eSSDs110sdo not receive Namespace create/delete commands directly. Normally, the S-eSSDs110swould contain a single Namespace representing the entire capacity. When appropriate, the P-eSSD110pmay issue a Namespace Create command or a Delete command to the S-eSSDs110sas sub-commands132. The S-eSSDs110sthen respectively execute those commands, and send a corresponding completion entry134back to the P-eSSD110p. This flow is the same as for any such Admin sub-command it may receive from P-eSSD110p.

FIG. 9depicts a flowchart900depicting the execution of a Read/Write command under control of the P-eSSD110p, according to an embodiment of the present disclosure.

Referring toFIG. 9, the P-eSSD110pmay receive and execute all of the I/O commands180, including Read and Write commands180. When the P-eSSD110preceives a Read/Write command180(S910), the P-eSSD110pfirst looks up the MAT300(S920). From the MAT300, the P-eSSD110pidentifies the set of eSSDs where associated user data is located.

As described with respect toFIG. 6, the P-eSSD110pthen creates a command context for the original command (S930) so that the P-eSSD110pcan track execution of the sub-commands132. The P-eSSD110pthen creates the corresponding Read/Write sub-commands132(S940), and sends appropriate sub-commands132to the appropriate S-eSSDs110s(S950). The P-eSSD110palso provides all of the necessary transport network related information (e.g., addresses) to the S-eSSDs110s. As part of the sub-commands132, the S-eSSDs110sreceive remote host190SGLs that contain remote buffer address, size, and security key.

The data transfer fields in the sub-commands132are appropriately modified to the right offsets. The S-eSSDs110sperform direct data transfers to the remote host190buffers (S960), and when complete, the S-eSSDs110ssend the completions to the P-eSSD110p(as opposed to directly to the host190). If necessary, the P-eSSD110pperforms its own share of data transfer to the remote host190(S960).

Furthermore, each of the S-eSSDs110sreceives sufficient information from the P-eSSD110pto perform any direct data transfers to the remote host190(S960). In the NVMe-oF protocol, RDMA transport services (RDMA Read and RDMA Write) are used for the data transfers from the S-eSSDs110sto the remote host190. The remote host190may need to support the Shared Receive Queue (SRQ) feature of RDMA protocol to allow multiple S-eSSDs110sto transfer data to the remote host190(S960). RDMA protocol may run over Ethernet/IP/TCP (iWARP), Ethernet/InfiniBand (RoCE v1), or Ethernet/IP/UDP (RoCE v2). With respect to communication between the S-eSSDs110sand the remote host190, the S-eSSDs110sstrictly perform only data transfers with the remote host190(S960). That is, only RDMA-Read and RDMA-Write operations are performed by the S-eSSDs110swith the remote host190(S960). The sub-command completion entries134and any non-user data transfers are performed with the P-eSSD110pusing an RDMA Send operation or some other proprietary protocol.

When all of the sub-commands are completed (S970), as indicated when all of the sub-command completion entries are received by the P-eSSD110p, the P-eSSD110pcreates a completion entry182for the original command180, and sends the completion entry182to an appropriate CQ on the host190(S980). The P-eSSD110pthen deallocates the command context (S990).

FIG. 10depicts a flowchart1000depicting Admin sub-command execution by an S-eSSD110s, according to an embodiment of the present disclosure.

Referring toFIG. 10, in the present embodiment, no S-eSSD110sever receives an Admin command180, or any command, directly from the host190NVMe-oF driver170. Instead, the P-eSSD110psends an Admin sub-command132, only when necessary, to the S-eSSDs110s(S1010). The S-eSSDs110sthen determines whether any data transfer is needed (S1020), and then perform any data transfers needed with P-eSSD110pover the private communications channel130(S1020). The S-eSSDs110sthen create (S1040) and send a completion entry134to the P-eSSD110p(S1050). In other embodiments, a S-eSSD110scan use an RDMA Send operation to send data, as well as completion entries134, to the P-eSSD110p.

FIG. 11depicts a flowchart1100depicting a Read/Write sub-command execution by the S-eSSD110s, according to an embodiment of the present disclosure.

Referring toFIG. 11, the S-eSSDs primarily work on Read or Write sub-commands132(e.g., as opposed to Admin sub-commands132). That is, the S-eSSDs110sprimarily perform data movements to and from remote hosts190without getting into other aspects of the protocol processing. When the S-eSSD110sreceives a Read/Write sub-command (S1110), the S-eSSD110suses the received transport network information (S1120) to issue RDMA Read or RDMA Write requests to the remote host190(S1130). As part of the sub-commands132, the S-eSSD110sreceives the details of the remote buffer address/offset, size, and security key. When the necessary data transfer is complete (S1140), the S-eSSD110ssends the completion entry134with appropriate error status to the P-eSSD110p(S1150).

FIG. 12depicts a flowchart depicting data transfer and sub-command completion synchronization in S-eSSDs110S, according to an embodiment of the present disclosure.

Referring toFIG. 12, for a given host command180, although the P-eSSD110psends the completion entry182to the host190, a set of the S-eSSDs110smay perform data transfers to the host190. All of the data should be transferred to a remote host190before a completion entry182for that given command180is provided to the host190, as a completion entry182for a command180reaching the host190before the associated data may result in un-defined behavior/an error.

As described above, the P-eSSD110preceives a Read/Write command180from the host (S1210), and segments the command180into a plurality of sub-commands132(S1220). Then, each of the S-eSSDs110sreceiving a respective sub-command132from the P-eSSD issues a data transfer to the host (S1250), and upon determining that the data transfer is complete (S1260), the S-eSSD110ssends a completion entry134to the P-eSSD110p(S1270). Then, upon receiving all of the sub-command completion entries134from every relevant S-eSSD110s(S1230), the P-eSSD110psends a completion entry182to the host (S1240).

Because Aggregated Ethernet SSD data transfers and completion entry posting are distributed over a set of eSSDs110, data and completion synchronization should be achieved. Usually, when a single eSSD performs both of the phases of a command execution (data transfer+completion posting), such problems may not arise. However, that is not the case with Aggregated Ethernet SSD. Accordingly, the P-eSSD110pmust wait for all of the sub-command completion entries134from the respective S-eSSDs110sbefore posting a completion entry182for a command180. Further, the S-eSSDs110smust ensure that all of their data transfers are completely and reliably finished before sending the completion entry134of the sub-commands132to the P-eSSD110p. Such two-step synchronization process ensures that the NVMe-oF protocol integrity is achieved in an Aggregated Ethernet SSD system all the time.

According to the above, because only a single primary eSSD is visible to the host, and performs all of the NVMe-oF protocol processing while tracking completion of all relevant sub-commands by the secondary eSSDs, the aggregation group of eSSDs is presented as a single, large logical capacity to the host.