System and method for lockless aborting of input/output (IO) commands

A method, computer program product, and computing system for receiving an input/output (IO) command for processing data within a storage system. An IO command-specific entry may be generated in a register based upon, at least in part, the IO command. An compare-and-swap operation may be performed on the IO command-specific entry to determine an IO command state associated with the IO command. The IO command may be processed based upon, at least in part, the IO command state associated with the IO command.

BACKGROUND

Storing and safeguarding electronic content may be beneficial in modern business and elsewhere. Accordingly, various methodologies may be employed to communicate data between storage processors and storage arrays more efficiently.

Many storage systems utilize non-volatile memory express (NVMe) devices (i.e., NVMe drives or storage devices that support NVMEoF connectivity) according to the NVMe storage protocol and/or other storage protocols. Such storage protocols allow for IO command aborting/termination by issuing a subsequent IO command configured to instruct the storage system to abort or terminate the specified IO command. In the specific case of NVMe, an NVMe IO command may be submitted for processing on a particular IO submission queue of a specific storage controller. To abort a particular NVMe IO command, an NVMe IO abort command may be issued to the same storage controller that the NVMe IO command was submitted to. Conventional approaches to aborting NVMe IO commands require some lock over the command. The completion flow and the aborting flow will take the lock and thus the command state will be consistent with the applied logic. However, this approach hurts performance by requiring computationally heavy “lock” and “unlock” operations for every IO command even if it is not being aborted, and when an IO command is being aborted, it is undesirable to just “spin” in the completion flow.

SUMMARY OF DISCLOSURE

In one example implementation, a computer-implemented method executed on a computing device may include but is not limited to receiving an input/output (IO) command for processing data within a storage system. An IO command-specific entry may be generated in a register based upon, at least in part, the IO command. An compare-and-swap operation may be performed on the IO command-specific entry to determine an IO command state associated with the IO command. The IO command may be processed based upon, at least in part, the IO command state associated with the IO command.

One or more of the following example features may be included. Generating the IO command-specific entry in a register based upon, at least in part, the IO command includes generating the IO command-specific entry with one or more IO command parameters and a default IO command state. The storage system includes a non-volatile memory express (NVMe) storage system. The one or more IO parameters include one or more of: an NVMe Subsystem identifier; an NVMe controller identifier; an NVMe submission queue identifier; and an NVMe submission queue command identifier. The IO command state includes an IO command abort state. performing the compare-and-swap operation on the IO command-specific entry to determine the IO command state associated with the IO command includes one or more of: determining that the IO command abort state associated with the IO command is abortable; determining that the IO command abort state associated with the IO command is aborted; determining that the IO command abort state associated with the IO command is aborting; and determining that the IO command abort state associated with the IO command is complete. Processing the IO command based upon, at least in part, the IO command state associated with the IO command includes modifying the IO command abort state associated with the IO command based upon, at least in part, the compare-and-swap operation on the IO command-specific entry.

In another example implementation, a computer program product resides on a computer readable medium that has a plurality of instructions stored on it. When executed by a processor, the instructions cause the processor to perform operations that may include but are not limited to receiving an input/output (IO) command for processing data within a storage system. An IO command-specific entry may be generated in a register based upon, at least in part, the IO command. An compare-and-swap operation may be performed on the IO command-specific entry to determine an IO command state associated with the IO command. The IO command may be processed based upon, at least in part, the IO command state associated with the IO command.

One or more of the following example features may be included. Generating the IO command-specific entry in a register based upon, at least in part, the IO command includes generating the IO command-specific entry with one or more IO command parameters and a default IO command state. The storage system includes a non-volatile memory express (NVMe) storage system. The one or more IO parameters include one or more of: an NVMe Subsystem identifier; an NVMe controller identifier; an NVMe submission queue identifier; and an NVMe submission queue command identifier. The IO command state includes an IO command abort state. performing the compare-and-swap operation on the IO command-specific entry to determine the IO command state associated with the IO command includes one or more of: determining that the IO command abort state associated with the IO command is abortable; determining that the IO command abort state associated with the IO command is aborted; determining that the IO command abort state associated with the IO command is aborting; and determining that the IO command abort state associated with the IO command is complete. Processing the IO command based upon, at least in part, the IO command state associated with the IO command includes modifying the IO command abort state associated with the IO command based upon, at least in part, the compare-and-swap operation on the IO command-specific entry.

In another example implementation, a computing system includes at least one processor and at least one memory architecture coupled with the at least one processor, wherein the at least one processor is configured to perform operations that may include but are not limited to receiving an input/output (IO) command for processing data within a storage system. The processor may be further configured to generate an IO command-specific entry in a register based upon, at least in part, the IO command. The processor may be further configured to perform an compare-and-swap operation on the IO command-specific entry to determine an IO command state associated with the IO command. The processor may be further configured to process the IO command based upon, at least in part, the IO command state associated with the IO command.

One or more of the following example features may be included. Generating the IO command-specific entry in a register based upon, at least in part, the IO command includes generating the IO command-specific entry with one or more IO command parameters and a default IO command state. The storage system includes a non-volatile memory express (NVMe) storage system. The one or more IO parameters include one or more of: an NVMe Subsystem identifier; an NVMe controller identifier; an NVMe submission queue identifier; and an NVMe submission queue command identifier. The IO command state includes an IO command abort state. performing the compare-and-swap operation on the IO command-specific entry to determine the IO command state associated with the IO command includes one or more of: determining that the IO command abort state associated with the IO command is abortable; determining that the IO command abort state associated with the IO command is aborted; determining that the IO command abort state associated with the IO command is aborting; and determining that the IO command abort state associated with the IO command is complete. Processing the IO command based upon, at least in part, the IO command state associated with the IO command includes modifying the IO command abort state associated with the IO command based upon, at least in part, the compare-and-swap operation on the IO command-specific entry.

DETAILED DESCRIPTION

Referring toFIG.1, there is shown IO command aborting process10that may reside on and may be executed by storage system12, which may be connected to network14(e.g., the Internet or a local area network). Examples of storage system12may include, but are not limited to: a Network Attached Storage (NAS) system, a Storage Area Network (SAN), a personal computer with a memory system, a server computer with a memory system, and a cloud-based device with a memory system.

As is known in the art, a SAN may include one or more of a personal computer, a server computer, a series of server computers, a mini computer, a mainframe computer, a RAID device and a NAS system. The various components of storage system12may execute one or more operating systems, examples of which may include but are not limited to: Microsoft® Windows®; Mac® OS X®; Red Hat® Linux®, Windows® Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Mac and OS X are registered trademarks of Apple Inc. in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both).

The instruction sets and subroutines of IO command aborting process10, which may be stored on storage device16included within storage system12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage system12. Storage device16may include but is not limited to: a hard disk drive; a tape drive; an optical drive; a RAID device; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices (e.g., Solid State memory storage devices). Additionally/alternatively, some portions of the instruction sets and subroutines of IO command aborting process10may be stored on storage devices (and/or executed by processors and memory architectures) that are external to storage system12.

Various IO requests (e.g. IO request20) may be sent from client applications22,24,26,28to storage system12. Examples of IO request20may include but are not limited to data write requests (e.g., a request that content be written to storage system12) and data read requests (e.g., a request that content be read from storage system12).

The instruction sets and subroutines of client applications22,24,26,28, which may be stored on storage devices30,32,34,36(respectively) coupled to client electronic devices38,40,42,44(respectively), may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices38,40,42,44(respectively). Storage devices30,32,34,36may include but are not limited to: hard disk drives; tape drives; optical drives; RAID devices; random access memories (RAM); read-only memories (ROM), and all forms of flash memory storage devices (e.g., Solid State memory storage devices). Examples of client electronic devices38,40,42,44may include, but are not limited to, personal computer38, laptop computer40, smartphone42, notebook computer44, a server (not shown), a data-enabled, cellular telephone (not shown), and a dedicated network device (not shown).

Users46,48,50,52may access storage system12directly through network14or through secondary network18. Further, storage system12may be connected to network14through secondary network18, as illustrated with link line54.

Client electronic devices38,40,42,44may each execute an operating system, examples of which may include but are not limited to Microsoft® Windows®; Mac® OS X®; Red Hat® Linux®, Windows® Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Mac and OS X are registered trademarks of Apple Inc. in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both).

In some implementations, as will be discussed below in greater detail, a process, such as IO command aborting process10ofFIG.1, may include but is not limited to, receiving an input/output (IO) command for processing data within a storage system. An IO command-specific entry may be generated in a register based upon, at least in part, the IO command. An compare-and-swap operation may be performed on the IO command-specific entry to determine an IO command state associated with the IO command. The IO command may be processed based upon, at least in part, the IO command state associated with the IO command.

For example purposes only, storage system12will be described as being a network-based storage system that includes a plurality of electro-mechanical backend storage devices. However, this is for example purposes only and is not intended to be a limitation of this disclosure, as other configurations are possible and are considered to be within the scope of this disclosure.

The Storage System:

Referring also toFIG.2, storage system12may include storage processor100and a plurality of storage targets T 1-n (e.g., storage targets102,104,106,108). Storage targets102,104,106,108may be configured to provide various levels of performance and/or high availability. For example, one or more of storage targets102,104,106,108may be configured as a RAID 0 array, in which data is striped across storage targets. By striping data across a plurality of storage targets, improved performance may be realized. However, RAID 0 arrays do not provide a level of high availability. Accordingly, one or more of storage targets102,104,106,108may be configured as a RAID 1 array, in which data is mirrored between storage targets. By mirroring data between storage targets, a level of high availability is achieved as multiple copies of the data are stored within storage system12.

While storage targets102,104,106,108are discussed above as being configured in a RAID 0 or RAID 1 array, this is for example purposes only and is not intended to be a limitation of this disclosure, as other configurations are possible. For example, storage targets102,104,106,108may be configured as a RAID 3, RAID 4, RAID 5 or RAID 6 array.

While in this particular example, storage system12is shown to include four storage targets (e.g. storage targets102,104,106,108), this is for example purposes only and is not intended to be a limitation of this disclosure. Specifically, the actual number of storage targets may be increased or decreased depending upon e.g., the level of redundancy/performance/capacity required.

Storage system12may also include one or more coded targets110. As is known in the art, a coded target may be used to store coded data that may allow for the regeneration of data lost/corrupted on one or more of storage targets102,104,106,108. An example of such a coded target may include but is not limited to a hard disk drive that is used to store parity data within a RAID array.

While in this particular example, storage system12is shown to include one coded target (e.g., coded target110), this is for example purposes only and is not intended to be a limitation of this disclosure. Specifically, the actual number of coded targets may be increased or decreased depending upon e.g. the level of redundancy/performance/capacity required.

Examples of storage targets102,104,106,108and coded target110may include one or more electro-mechanical hard disk drives and/or solid-state/flash devices, wherein a combination of storage targets102,104,106,108and coded target110and processing/control systems (not shown) may form data array112.

The manner in which storage system12is implemented may vary depending upon e.g. the level of redundancy/performance/capacity required. For example, storage system12may be a RAID device in which storage processor100is a RAID controller card and storage targets102,104,106,108and/or coded target110are individual “hot-swappable” hard disk drives. Another example of such a RAID device may include but is not limited to an NAS device. Alternatively, storage system12may be configured as a SAN, in which storage processor100may be e.g., a server computer and each of storage targets102,104,106,108and/or coded target110may be a RAID device and/or computer-based hard disk drives. Further still, one or more of storage targets102,104,106,108and/or coded target110may be a SAN.

In the event that storage system12is configured as a SAN, the various components of storage system12(e.g. storage processor100, storage targets102,104,106,108, and coded target110) may be coupled using network infrastructure114, examples of which may include but are not limited to an Ethernet (e.g., Layer 2 or Layer 3) network, a fiber channel network, an InfiniBand network, or any other circuit switched/packet switched network. As will be discussed in greater detail below and in some implementations, network infrastructure114may include one or more storage fabrics. A storage fabric may generally include switches, routers, protocol bridges, gateway devices, and cables configured to connect components of storage system12.

Storage system12may execute all or a portion of IO command aborting process10. The instruction sets and subroutines of IO command aborting process10, which may be stored on a storage device (e.g., storage device16) coupled to storage processor100, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage processor100. Storage device16may include but is not limited to: a hard disk drive; a tape drive; an optical drive; a RAID device; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices (e.g., Solid State memory storage devices). As discussed above, some portions of the instruction sets and subroutines of IO command aborting process10may be stored on storage devices (and/or executed by processors and memory architectures) that are external to storage system12.

As discussed above, various IO requests (e.g. IO request20) may be generated. For example, these IO requests may be sent from client applications22,24,26,28to storage system12. Additionally/alternatively and when storage processor100is configured as an application server, these IO requests may be internally generated within storage processor100. Examples of IO request20may include but are not limited to data write request116(e.g., a request that content120be written to storage system12) and data read request118(i.e. a request that content120be read from storage system12).

During operation of storage processor100, content120to be written to storage system12may be processed by storage processor100. Additionally/alternatively and when storage processor100is configured as an application server, content120to be written to storage system12may be internally generated by storage processor100.

Storage processor100may include frontend cache memory system122. Examples of frontend cache memory system122may include but are not limited to a volatile, solid-state, cache memory system (e.g., a dynamic RAM cache memory system) and/or a non-volatile, solid-state, cache memory system (e.g., a flash-based, cache memory system or a non-volatile dual in-line memory module (NVDIMM-N)).

Storage processor100may initially store content120within frontend cache memory system122. Depending upon the manner in which frontend cache memory system122is configured, storage processor100may immediately write content120to storage array112(if frontend cache memory system122is configured as a write-through cache) or may subsequently write content120to storage array112(if frontend cache memory system122is configured as a write-back cache).

Storage array112may include backend cache memory system124. Examples of backend cache memory system124may include but are not limited to a volatile, solid-state, cache memory system (e.g., a dynamic RAM cache memory system) and/or a non-volatile, solid-state, cache memory system (e.g., a flash-based, cache memory system or a NVDIMM-N system). During operation of storage array112, content120to be written to storage array112may be received from storage processor100. Storage array112may initially store content120within backend cache memory system124prior to being stored on e.g. one or more of storage targets102,104,106,108, and coded target110.

As discussed above, the instruction sets and subroutines of IO command aborting process10, which may be stored on storage device16included within storage system12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage system12. Accordingly, in addition to being executed on storage processor100, some or all of the instruction sets and subroutines of IO command aborting process10may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage array112.

Further and as discussed above, during the operation of storage array112, content (e.g., content120) to be written to storage array112may be received from storage processor100and initially stored within backend cache memory system124prior to being stored on e.g. one or more of storage targets102,104,106,108,110. Accordingly, during use of storage array112, backend cache memory system124may be populated (e.g., warmed) and, therefore, subsequent read requests may be satisfied by backend cache memory system124(e.g., if the content requested in the read request is present within backend cache memory system124), thus avoiding the need to obtain the content from storage targets102,104,106,108,110(which would typically be slower).

In the context of storage systems, a storage processor (e.g., storage processor100) may include one or more central processing units (CPUs) with one or more cores, a cache memory system (e.g., cache memory system122), and one or more network interface cards (NICs). As discussed above and in some implementations, the storage processor (e.g., storage processor100) may be communicatively coupled with a storage array (e.g., storage array112). In some implementations, the storage array (e.g., storage array112) may include one or more storage devices. In some implementations, the storage array may be a non-volatile memory disk array with one or more solid-state drives (SSDs). The storage processor100may interact with the one or more SSDs via the non-volatile memory express (NVMe™) protocol or standard. NVMe is a trademark of NVM Express, Inc. in the United States, other countries, or both. In this manner, storage devices, such as SSDs, that are configured to communicate with a storage processor via the NVMe protocol may generally be referred to as NVMe devices.

As is known in the art, non-volatile memory express (NVMe) may generally include a host controller interface and storage protocol configured to transfer data between client systems and SSDs over a Peripheral Component Interconnect Express (PCIe) bus. Referring also to the example ofFIG.3and in some implementations, one or more storage processors (e.g., storage processor100and storage processor300) may be configured to be communicatively coupled to a storage array (e.g., storage array112) via one or more storage fabrics (e.g., storage fabrics302,304). In some implementations, storage fabrics may be internal to storage system12and/or may be shared with clients (e.g., as shown in network14ofFIG.1). In this manner and as will be discussed in greater detail below, the storage system (e.g., storage system12) may include a plurality of storage processors configured to receive a plurality of IO requests (e.g., write requests, read requests, etc.).

In some implementations, storage processors100,300may be communicatively coupled with storage array112via a non-volatile memory express over fabrics protocol. NVMe-oF is a trademark of NVM Express, Inc. in the United States, other countries, or both. NVM Express over Fabrics (NVMe-oF) may generally define a common architecture that supports a range of storage networking fabrics for NVMe block storage protocol over a storage networking fabric. This includes enabling a front-side interface into storage systems, scaling out to large numbers of NVMe devices and extending the distance within a datacenter over which NVMe devices and NVMe subsystems can be accessed.

In some implementations, storage processors100,300may be communicatively coupled to storage array112via one or more storage fabrics (e.g., storage fabrics302,304). In some implementations, storage fabric302may be a primary storage fabric while storage fabric304may be a high-availability or duplicate storage fabric in the event of a failure of storage fabric302. In some implementations, storage fabrics302,304may be used by storage processors100,300to access storage array112. While the example ofFIG.3includes e.g., two storage fabrics, it will be appreciated that any number of storage fabrics may be used within the scope of the present disclosure to communicatively couple storage processors100,300and storage array112.

In some implementations, storage array112may include one or more storage array enclosures. In some implementations, a storage array enclosure (e.g., storage array enclosure306) may generally include a drive carrier, one or more disk drives, a midplane, one or more storage controllers, and/or one or more persistent memory devices. In some implementations, the driver carrier of storage array enclosure306may be configured to hold one or more disk drives (e.g., NVMe drives308,310,312). In some implementations, NVMe devices may include dual-ported NVMe devices configured to be communicatively coupled to two storage controllers. In this manner, failure of one storage controller may not impact access to each NVMe device. Storage array enclosure306may include a midplane configured to distribute power and signals to components within the storage array enclosure.

In some implementations, storage controllers (e.g., storage controllers314,316) may generally include a module configured to support, control, and monitor the storage array enclosure. In some implementations, storage controllers314,316may include various input and/or output connectors for coupling one or more storage processors (e.g., storage processors100,300) to one or more disk drives (e.g., NVMe drives308,310,312). In some implementations, storage array enclosure306may include dual controllers configured to provide access to the NVMe devices of storage array enclosure306. In this manner, the dual controllers may provide no single point of failure for access to the NVMe devices of storage array112.

The Storage System Memory Architecture:

In the context of storage systems, a storage processor (e.g., storage processor100) may include one or more central processing units (CPUs) with one or more cores, a cache memory system (e.g., cache memory system122), and one or more network interface cards (NICs). As discussed above and in some implementations, the storage processor (e.g., storage processor100) may be communicatively coupled with a storage array or data array (e.g., data array112). In some implementations, the data array (e.g., data array112) may include one or more storage devices. In some implementations, the data array may be a non-volatile memory disk array with one or more solid-state drives (SSDs). The storage processor100may interact with the one or more SSDs via the non-volatile memory express (NVMe) protocol or standard. In this manner, storage devices, such as SSDs, that are configured to communicate with a storage processor via the NVMe protocol may generally be referred to as NVMe storage devices.

As is known in the art, NVMe may generally include a host controller interface and storage protocol configured to transfer data between client systems and SSDs over a Peripheral Component Interconnect Express (PCIe) bus. Referring also to the example ofFIG.4and in some implementations, the storage system memory architecture (e.g., within the memory system122of storage processor100) may be represented a user space layer (e.g., user space layer400) and a kernel layer (e.g., kernel layer402). A user space layer (e.g., user space layer400) may generally include a set of memory locations in which one or more threads (e.g., threads404,406,408) are executed. A thread (e.g., threads404,406,408) may generally include at least a portion of an executing instance of a program or application. For example and in some implementations, one or more client electronic devices (e.g., client electronic devices38,40,42,44) may execute one or more programs or applications with data stored in a storage system (e.g., storage system12) via a storage processor (e.g., storage processor100). In some implementations, threads404,406,408may store user data (e.g., from client electronic devices38,40,42,44) and its metadata on a data array (e.g., data array112) of storage system12. As discussed above, the data array may include one or more NVMe storage devices (e.g., NVMe drives308,310,312).

In some implementations, the kernel (e.g., kernel402) may generally include the set of memory locations where the kernel of an operating system (OS) executes or runs and/or where one or more device drivers may execute. As is known in the art, the kernel may manage individual threads within the user space to prevent them from interfering with one another as they communicate data to and from the data array (e.g., data array112). In some implementations, kernel402may be accessed by threads404,406,408within user space400by the use of system calls. Examples of system calls may include IO requests or operations configured to transfer data between a CPU and a peripheral device (e.g., one or more NVMe storage devices). However, it will be appreciated that other system calls are possible within the scope of the present disclosure.

In some implementations, kernel402may include an NVMe kernel driver (e.g., a NVMe kernel driver410) deployed in kernel402for communicating data between threads404,406,408of user space400and one or more NVMe storage devices (e.g., NVMe drives308,310,312). In some implementations, the NVMe kernel driver (e.g., NVMe kernel driver410) may be a standard Linux® NVMe kernel driver generally available in most storage processors. In some implementations, threads404,406,408may communicate with NVMe kernel driver410via a block interface (e.g., block interface412).

In some implementations, NVMe kernel driver410may include one or more IO submission queues (e.g., IO submission queues414,416,418) and one or more IO completion queues (e.g., IO completion queues420,422,424). In some implementations, IO submission queues414,416,418and IO completion queues420,422,424may be organized in pairs (e.g., IO submission queue414paired with IO completion queue420). IO submission queue414may generally send one or more IO requests (e.g., IO request20) to the one or more NVMe storage devices (e.g., NVMe drives308,310,312) and IO completion queue420may generally receive a completion for a corresponding IO request of IO submission queue414. In some implementations, the one or more IO submission queues and the one or more completion queues may be organized per core of a multi-core CPU, per NVMe storage device, and/or per core and per NVMe storage device (e.g., one or more IO submission queues and one or more IO completion queues for each core and NVMe storage device). In the example ofFIG.4, suppose for example purposes only that the CPU of storage processor100is a single core CPU. In this example, a pair of IO submission queues and IO completion queues may be created by NVMe kernel driver410for each NVMe storage device (e.g., NVMe drives308,310,312) of a data array (e.g., data array112). However, it will be appreciated that any number of CPU cores and/or NVMe storage devices may be used within the scope of the present disclosure.

In some implementations, NVMe kernel driver410may include one or more administrative IO submission queues (e.g., IO submission queue426) and one or more administrative IO completion queues (e.g., IO completion queue428) which may or may not be organized in pairs. Administrative IO submission queue426may be configured to provide one or more management operations (e.g., create and/or destroy IO submission queues, create and/or destroy IO completion queues, etc.) and administrative IO completion queue428may be configured to receive completions for a corresponding IO request of administrative IO submission queue426.

The IO Command Aborting Process:

Referring also toFIG.5and in some implementations, IO command aborting process10may receive500an input/output (IO) command for processing data within a storage system. An IO command-specific entry may be generated502in a register based upon, at least in part, the IO command. An compare-and-swap operation may be performed504on the IO command-specific entry to determine an IO command state associated with the IO command. The IO command may be processed506based upon, at least in part, the IO command state associated with the IO command.

As will be discussed in greater detail below, implementations of the present disclosure may allow a lockless approach to IO command aborting in a multithreaded storage system. For example, many storage systems utilize non-volatile memory express (NVMe) devices (i.e., NVMe drives or storage devices that support NVMEoF connectivity) according to the NVMe storage protocol and/or other storage protocols. Such storage protocols allow for IO command aborting/termination by issuing a subsequent IO command configured to instruct the storage system to abort or terminate the specified IO command. In the specific case of NVMe, an NVMe IO command may be submitted for processing on a particular IO submission queue of a specific storage controller. To abort a particular NVMe IO command, an NVMe IO abort command may be issued to the same storage controller that the NVMe IO command was submitted to. Conventional approaches to allowing for NVMe IO commands require some lock over the command. The completion flow and the abort flow will take the lock and thus the command state will be consistent with the applied logic. The problem with this approach is that it hurts performance: “lock” and “unlock” operations are heavy and are required for every IO command even if it is not being aborted, and when an IO command is being aborted, it is undesirable to just “spin” in the completion flow. Accordingly and as will be discussed in greater detail below, IO command aborting process10may provide state consistency without using any locking mechanism(s).

In some implementations, IO command aborting process10may receive500an input/output (IO) command for processing data within a storage system. For example and as discussed above, IO commands may include but are not limited to data write request116(e.g., a request that content120be written to storage system12) and data read request118(i.e. a request that content120be read from storage system12). Referring also to FIG.5, storage system12may receive500an IO command (e.g., IO command600) for processing. In this example, IO command600may be configured to read data from or write data to one or more NVMe devices (e.g., NVMe drives308,310,312) of a data array (e.g., data array112). As will be discussed in greater detail below, it may be desirable to abort or terminate the execution of the IO command.

The storage system may include a non-volatile memory express (NVMe) storage system. As discussed above, a non-volatile memory express (NVMe) storage system may generally include a host controller interface and storage protocol configured to transfer data between client systems and SSDs over a Peripheral Component Interconnect Express (PCIe) bus. Referring again to the example ofFIG.3and in some implementations, one or more storage processors (e.g., storage processor100and storage processor300) may be configured to be communicatively coupled to a storage array (e.g., storage array112) via one or more storage fabrics (e.g., storage fabrics302,304). The storage processors (e.g., storage processor100and storage processor300) may interact with the one or more storage controllers (e.g., storage controllers314,316). In this manner, storage devices, such as SSDs, that are configured to communicate with a storage processor via the NVMe protocol may generally be referred to as NVMe storage devices (e.g., NVMe drives308,310,312).

In some implementations, IO command aborting process10may generate502an IO command-specific entry in a register based upon, at least in part, the IO command. An IO command-specific entry may generally include a bespoke identifier for each IO command. Referring again to the example ofFIG.6, IO command aborting process10may receive500an IO command (e.g., IO command600) and may generate502an IO command-specific entry (e.g., IO command-specific entry602) and store the IO command-specific entry (e.g., IO command-specific entry602) within a register (e.g., register604). A register (e.g., register604) may generally include a type of computer memory used to quickly accept, store, and transfer data and instructions that are being used immediately by a central processing unit (CPU) of a storage processor (e.g., storage processor100). Register604may be communicatively coupled to and/or integrated within a storage processor (e.g., storage processor100).

Generating502the IO command-specific entry in a register based upon, at least in part, the IO command may include generating508the IO command-specific entry with one or more IO command parameters and a default IO command state. For example, the IO command-specific entry (e.g., IO command-specific entry602) may be defined to be a unique or bespoke representation of the IO command (e.g., IO command600) using one or more IO command parameters associated with the IO command (e.g., IO command600).

In one example, the one or more IO parameters may include one or more of: an NVMe Subsystem identifier; an NVMe controller identifier; an NVMe submission queue identifier; and an NVMe submission queue command identifier. For example, each storage system (e.g., storage system12) may include one or more NVMe Subsystems. Referring again toFIG.3, storage system12may include one NVMe Subsystem. However, it will be appreciated that storage system12may include any number of NVMe Subsystems within the scope of the present disclosure. In this manner, the one or more IO parameters may include a reference to each NVMe Subsystem such that each NVMe Subsystem may be uniquely identified.

Referring again toFIG.3, the storage system (e.g., storage system12) may include one or more storage controllers (e.g., storage controllers314,316). In this example, IO command aborting process10may generate502an IO command-specific entry (e.g., IO command-specific entry602) with an NVMe controller identifier that specifies a particular storage controller within the storage system (e.g., storage system12).

Referring again toFIG.4, each storage controller (e.g., storage controllers314,316) may include an NVMe kernel driver (e.g., NVMe kernel driver410) configured to process IO commands on the plurality of NVMe devices (e.g., NVMe drives308,310,312). As discussed above, each NVMe kernel driver (e.g., NVMe kernel driver410) may include one or more IO submission queues (e.g., IO submission queues414,416,418). In this example, IO command aborting process10may generate502an NVMe submission queue identifier associated with an IO command (e.g., IO command600). In this manner, the particular NVMe IO submission queue (e.g., IO submission queues414,416,418) of a storage controller (e.g., storage controllers314,316) may be uniquely identified.

In some implementations, each IO command of an IO submission queue may have a particular IO command identifier. In this example, IO command aborting process10may generate502an IO command identifier associated with an IO command (e.g., IO command600). In this manner, the particular IO command stored in an IO submission queue (e.g., IO submission queues414,416,418) may be uniquely identified.

In some implementations, IO command aborting process10may generate502the IO command-specific entry (e.g., IO command-specific entry602) using the combination of one or more of the NVMe Subsystem identifier; the NVMe controller identifier; the NVMe submission queue identifier; the NVMe submission queue command identifier; and/or a default IO command state. The default IO command state may generally include an initial IO command state. In one example and as will be discussed in greater detail below, this initial IO command state may indicate that the IO command-specific entry is free. However, it will be appreciated that the default or initial IO command state may be defined as any value within the scope of the present disclosure.

In some implementations, the one or more IO parameters may be reused as IO commands are processed. For example, suppose that an IO command is processed from a particular IO submission queue of a specific storage controller. In this example, once the IO command is processed, certain IO parameters of the IO command-specific entry (e.g., IO command-specific entry602) may be reused. However, to address this situation, IO command aborting process10may generate508the IO command-specific entry (e.g., IO command-specific entry602) by combining the one or more IO command parameters and the default IO command state. In this example and as will be discussed in greater detail below, the IO command state may be used, in combination with the one or more IO command parameters, to uniquely identify the IO command; even if the one or more IO parameters are reused after processing a previous IO command.

In one example, the IO command-specific entry (e.g., IO command-specific entry602) may include a number of bits associated with an NVMe Subsystem identifier (e.g., 1 bit); a number of bits associated with an NVMe controller identifier (e.g., 16 bits); a number of bits associated with a particular NVMe submission queue (e.g., 16 bits); a number of bits associated with a particular IO command within the submission queue (e.g., 16 bits); and a number of bits associated with a particular IO command state (e.g., 3 bits). It will be appreciated that the IO command-specific entry (e.g., IO command-specific entry602) may include any number of bits within the scope of the present disclosure for various IO parameters and/or the IO command state.

In some implementations, IO command aborting process10may perform504a compare-and-swap operation on the IO command-specific entry to determine an IO command state associated with the IO command. As is known in the art, a compare-and-swap operation is an atomic instruction used in multithreading to achieve synchronization. The operation compares the contents of a memory location with a given value and, only if they are the same, modifies the contents of that memory location to a new given value. This is done as a single atomic operation. The atomicity guarantees that the new value is calculated based on up-to-date information; if the value had been updated by another thread in the meantime, the write would fail.

IO command aborting process10may perform504the compare-and-swap operation on the IO command-specific entry (e.g., IO command-specific entry602) to determine an IO command state. For example, if, during the compare-and-swap operation, one of the attributes of the IO command-specific entry (e.g., IO command-specific entry602) is changed, the compare-and-swap operation will fail. This may prevent the situation where an abort IO command is received for a particular IO command but the IO command has already completed and the IO command parameters are reused for another IO command. In conventional approaches, the original IO command would be locked upon receiving the abort IO command and/or the new IO command would be erroneously aborted. However, because a compare-and-swap operation may be performed on the entire register (i.e., all the bits of the register), the compare-and-swap operation will fail if the IO command was reused.

As the reading and writing of the IO command-specific entry (e.g., IO command-specific entry602) from the register (e.g., register604) is performed in a single memory operation, implementations of the present disclosure may reduce the number of memory operations needed to process abort IO commands within the storage system from the at least four memory operations required for locking and unlocking to only one memory operation.

In some implementations, the IO command state may include an IO command abort state. As discussed above, the IO command state may generally indicate the processing status of the IO command within the storage system. In one example, the IO command state may include the IO command's abort state or status. Examples of the IO command abort state include, but are not limited to, abortable; aborted; aborting; complete; and free. An abortable IO command may reference an IO command that is being processed but not yet complete. An aborting IO command may reference an IO command that is currently being aborted. An aborted IO command may reference an IO command that has been aborted. A completed IO command may reference an IO command that has been processed during the aborting state. While several example IO command abort states have been described, it will be appreciated that these are for example purposes only and that any number of or type of IO command abort state may be utilized within the scope of the present disclosure.

Performing504the compare-and-swap operation on the IO command-specific entry to determine the IO command state associated with the IO command may include one or more of: determining510that the IO command abort state associated with the IO command is abortable; determining512that the IO command abort state associated with the IO command is aborting; determining514that the IO command abort state associated with the IO command is aborted; and determining516that the IO command abort state associated with the IO command is complete. For example and as will be described below in greater detail below, IO command aborting process10may perform particular compare-and-swap operations to determine the IO command state. As will be discussed in greater detail below, by utilizing a compare-and-swap operation, IO command aborting process10may perform504an atomic operation to transition the IO command through various states or steps of processing without locking the IO command.

In some implementations, IO command aborting process10may process506the IO command based upon, at least in part, the IO command state associated with the IO command. Referring also the example ofFIG.7, the IO command abort state may be utilized to process the IO command according to the state diagram (e.g., state diagram700). For example and as discussed above, when an IO command (e.g., IO command600) is received500, IO command aborting process10may generate502an IO command-specific entry (e.g., IO command-specific entry602) for the IO command (e.g., IO command600) and may store the IO command-specific entry (e.g., IO command-specific entry602) in a register (e.g., register604). As shown inFIG.7, IO command aborting process10may transition the IO command abort state (e.g., shown as step “1”) to the free state (e.g., free state702). In response to generating502the IO command-specific entry (e.g., IO command-specific entry602), IO command aborting process10may perform504one or more compare-and-swap operations on the IO command specific entry (e.g., IO command-specific entry602) as shown in the exemplary pseudocode provided below:

As shown above in the example pseudocode and the state diagram ofFIG.7, IO command aborting process10may process506the IO command (e.g., IO command600) by transitioning the IO command abort state from the free state (e.g., free state702) to the abortable state (e.g., abortable state704). This transition is shown as step “2” inFIG.7. IO command aborting process10may then determine whether the IO command abort state is abortable (e.g., abortable state704). If the IO command abort state is determined510to be abortable (e.g., abortable state704), IO command aborting process10may perform504a compare-and-swap operation to determine if the IO command has completed. If so, IO command aborting process10may transition the IO command abort state from the free state (e.g., free state702) to the abortable state (e.g., abortable state704). This transition is shown as step “3” inFIG.7. If not, IO command aborting process10may remain in the abortable state (e.g., abortable stage704).

In response to determining510that the IO command abort state is abortable (e.g., abortable state704), IO command aborting process10may perform504a compare-and-swap operation to determine if the IO command is being aborted. For example and referring again to the example ofFIG.6, suppose IO command aborting process10receives an abort IO command (e.g., abort IO command606) that designates a particular IO command (e.g., IO command). In this example, IO command aborting process10may process the abort IO command (e.g., abort IO command606) to identify the IO command (e.g., IO command600). IO command aborting process10may determine proceed to abort the IO command (e.g., IO command600). In response to determining that the IO command is aborting, IO command aborting process10may transition the IO command abort state from the abortable state (e.g., abortable state704) to the aborting state (e.g., aborting state706). This transition is shown as step “4” inFIG.7. If not, IO command aborting process10may remain in the abortable state (e.g., abortable stage704).

In response to determining514that the IO command abort state is aborting (e.g., aborting state706), IO command aborting process10may perform504a compare-and-swap operation to determine if the IO command is aborted. If so, IO command aborting process10may transition the IO command abort state from the aborting state (e.g., aborting state706) to the aborted state (e.g., aborted state708). This transition is shown as step “5” inFIG.7. Additionally, IO command aborting process10may transition the IO command abort state from the aborting state (e.g., aborting state706) to the completed state (e.g., completed state710). This transition is shown as step “6” inFIG.7. In response to transitioning the IO command abort state to the aborted state (e.g., completed state708), IO command aborting process10may transition to the free state (e.g., free state702). This transition is shown as step “7” inFIG.7. Further and in response to transitioning the IO command abort state to the completed state (e.g., completed state710), IO command aborting process10may transition to the free state (e.g., free state702). This transition is shown as step “8” inFIG.7.

Processing506the IO command based upon, at least in part, the IO command state associated with the IO command may include modifying518the IO command abort state associated with the IO command based upon, at least in part, the one or more compare-and-swap operations on the IO command-specific entry. For example and as discussed above, when performing the one or more compare-and-swap operations on the IO command-specific entry (e.g., IO command-specific entry602), IO command aborting process10may modify518the IO command abort state associated with the IO command (e.g., IO command600). As discussed above, IO command aborting process10may utilize the results of each compare-and-swap operation to modify518the IO command abort state. For example, IO command aborting process10may update the IO command-specific entry (e.g., IO command-specific entry602) associated with an IO command (e.g., IO command600) in response to modifying518the IO command abort state.

In one example, suppose that IO command aborting process10is in the abortable state (e.g., abortable state704) and that IO command aborting process10performs504a compare-and-swap operation on the IO command-specific entry (e.g., IO command-specific entry602). In this example, IO command aborting process10may modify518the IO command abort state associated with the IO command (e.g., IO command600) from the abortable state (e.g., abortable state704) to the free state (e.g., free state702). Accordingly, IO command aborting process10may update the IO command-specific entry (e.g., IO command-specific entry602) with the modified IO command abort state (e.g., free state702).

In another example, suppose that IO command aborting process10is in the abortable state (e.g., abortable state704) and that IO command aborting process10performs504a compare-and-swap operation on the IO command-specific entry (e.g., IO command-specific entry602). In this example, IO command aborting process10may modify518the IO command abort state associated with the IO command (e.g., IO command600) from the abortable state (e.g., abortable state704) to the aborting state (e.g., aborting state706). Accordingly, IO command aborting process10may update the IO command-specific entry (e.g., IO command-specific entry602) with the modified IO command abort state (e.g., aborting state706).

Further suppose that IO command aborting process10is in the aborting state (e.g., aborting state706) and that IO command aborting process10performs504a compare-and-swap operation on the IO command-specific entry (e.g., IO command-specific entry602). In this example, IO command aborting process10may modify518the IO command abort state associated with the IO command (e.g., IO command600) from the aborting state (e.g., aborting state706) to the aborted state (e.g., aborted state708). Accordingly, IO command aborting process10may update the IO command-specific entry (e.g., IO command-specific entry602) with the modified IO command abort state (e.g., aborted state708).

In another example, suppose that IO command aborting process10is in the aborting state (e.g., aborting state706) and that IO command aborting process10performs a compare-and-swap operation on the IO command-specific entry (e.g., IO command-specific entry602). In this example, IO command aborting process10may modify518the IO command abort state associated with the IO command (e.g., IO command600) from the aborting state (e.g., aborting state706) to the completed state (e.g., completed state710). Accordingly, IO command aborting process10may update the IO command-specific entry (e.g., IO command-specific entry602) with the modified IO command abort state (e.g., completed state710).