Patent Description:
A host computing device may provide commands (e.g. read and write commands) for processing by a storage device. The host computing device may have certain quality of service (QoS) expectations for the processing of the commands. In certain situations, however, there may not be enough hardware resources to process the commands in compliance with the QoS requirements, resulting in jitter.

Accordingly, there is a need for a system and method for scheduling commands for processing by a storage device, that considers availability of the hardware resources of the storage device for addressing QoS requirements.

<NPL> discloses Storage Performance as a Managed Resource.

<CIT> discloses: A solid-state drive that can service multiple users or tenants and workloads (that is, multiple tenants) by enabling assigned bandwidth share of the solid-state drive across tenants. The assigned bandwidth share is enabled for command submissions within a same assigned domain in addition to a weighted bandwidth share and quality of service control across different domains from all tenants.

Embodiments of the present disclosure are set out in the appended claims.

An embodiment of the present disclosure is directed to a method for scheduling commands for processing by a storage device according to claim <NUM>. The method comprises receiving and storing a command from an application in a software queue; obtaining information on a hardware resources managed by the storage device; synchronizing a software resources based on the information on the hardware resources; allocating the software resources into a shared pool and a reserved pool; determining a condition of the software resources in the shared pool; allocating, to the command, one of the software resources in the shared pool, based on a first determination of the condition; and allocating, to the command, one of the software resources in the reserved pool, based on a second determination of the condition. Based on the allocating of the one of the software resources, the command is stored in a hardware queue associated with the storage device for processing by the storage device.

According to one embodiment, the command includes a data operation from an application running on the host, and the processing of the command includes performing the data operation with respect to the non-volatile storage medium.

According to one embodiment, the information on the hardware resources includes availability of the hardware resources for processing the command.

According to one embodiment, the synchronizing of the software resources includes determining satisfaction of a criterion; determining a number of available hardware resources based on satisfaction of the criterion; and setting a number of available software resources to be equal to a number of available hardware resources.

According to one embodiment, the criterion comprises an expiration of a time period.

According to one embodiment, the reserved pool is dynamically determined based on a detected criterion. The detected criterion may comprise an identification of a quality of service requirement for the software queue.

According to one embodiment, the method for scheduling commands for processing by a storage device further comprises allocating, by the storage device, one of the hardware resources based on storing the command in the hardware queue; processing the command by the storage device; and deallocating the one of the hardware resources based on completion of the processing of the command.

An embodiment of the present disclosure is also directed to a system for scheduling commands for processing by a storage device according to claim <NUM>. The system comprises a processor and a memory. The memory stores instructions that, when executed, cause the processor to: receive and store a command from an application in a software queue; obtain information on a hardware resources managed by the storage device; synchronize a software resources based on the information on the hardware resources; allocate the software resources into a shared pool and a reserved pool; determine a condition of the software resources in the shared pool; allocate, to the command, one of the software resources in the shared pool, based on a first determination of the condition; and allocate, to the command, one of the software resources in the reserved pool, based on a second determination of the condition, wherein, based on the allocating of the one of the software resources, the command is stored in a hardware queue associated with the storage device for processing by the storage device.

These and other features, aspects and advantages of the embodiments of the present disclosure will be more fully understood when considered with respect to the following detailed description, appended claims, and accompanying drawings. Of course, the actual scope of the invention is defined by the appended claims.

Non-limiting and non-exhaustive embodiments of the present embodiments are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

Hereinafter, example embodiments will be described in more detail with reference to the accompanying drawings, in which like reference numbers refer to like elements throughout. The present disclosure, however, may be embodied in various different forms, and should not be construed as being limited to only the illustrated embodiments herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of the present disclosure to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary to those having ordinary skill in the art for a complete understanding of the aspects and features of the present disclosure may not be described. Unless otherwise noted, like reference numerals denote like elements throughout the attached drawings and the written description, and thus, descriptions thereof may not be repeated. Further, in the drawings, the relative sizes of elements, layers, and regions may be exaggerated and/or simplified for clarity.

Generally speaking, a storage device may have a limited number of hardware resources for processing commands from a host computing device. The hardware resources may be, for example, hardware queues and associated tokens that may be used for scheduling the commands for processing. The size of the hardware queues and associated number of tokens may be implementation specific, and may depend on the storage device's performance and QoS requirements.

A device driver interacting with the storage device may allocate software resources to the commands that are to be processed. The software resources may be, for example, software queues and associated tokens. The depth of the software queues and associated tokens may be arbitrary, and exceed the depth of the hardware queues and its associated tokens. This may sometimes result in the device driver scheduling more commands than the hardware resources available in the storage device. In such a situation, a command that is scheduled by the device driver, but that cannot consume a hardware resource, may experience delays in processing, resulting in high jitter.

<FIG> is a conceptual block diagram of a system for scheduling commands according to current art mechanisms. The system includes a device driver <NUM> configured to submit commands for processing by a storage device <NUM>. The device driver <NUM> may include various software queues <NUM>-<NUM> for storing commands submitted by one or more applications of a host processing device. For example, queue <NUM> (Q1) may be a deep write queue configured to store write commands, while queue <NUM> (Q2) may be a shallow read queue configured to store read commands. The commands may consume software resources (e.g. software queues <NUM>-<NUM> and tokens <NUM>), to deliver the commands to the storage device <NUM> for processing.

Similar to the device driver <NUM>, the storage device <NUM> may include one or more hardware queues <NUM>-<NUM> configured to store the commands submitted by the device driver <NUM>. For example, hardware queue <NUM> may be configured to store commands fetched from Q1 <NUM> and Q2 <NUM>. The commands in the hardware queue <NUM> may consume hardware resources (e.g. hardware tokens) <NUM>. The commands with assigned hardware resources <NUM> may be scheduled for processing by the storage device.

The various software queues <NUM>-<NUM> configured in the device driver may for addressing QoS provisions for the processing of the commands. For example, by separating write and read commands into separate queues <NUM>, <NUM>, and by employing a scheduling algorithm such as round robin, the read commands in Q2 may generally be serviced in a predictable time window. In some situations, however, although there are commands in Q2 that have been queued up with the expectation of being processed, resource conflicts within the storage device may prevent the servicing of the commands in Q2 as expected.

To illustrate this point, an example may be considered where Q1 <NUM> has <NUM> write commands queued up, and Q2 <NUM> has <NUM> read commands queued up. It is assumed for purposes of this example that the <NUM> write commands consumes <NUM> software tokens <NUM>, and may be submitted to the storage device for processing. According to this example, however, the storage device <NUM> only has <NUM> hardware tokens <NUM> to be assigned. Thus, the storage device <NUM> fetches up to <NUM> write commands from Q1 <NUM> for processing, causing all <NUM> hardware tokens <NUM> to be consumed. Given that the write commands have consumed all the hardware tokens <NUM>, with more write commands with software tokens <NUM> assigned that are left to be processed, the read commands in Q2 may be unable to be processed during their expected time window. Thus, the QoS requirements for Q2 may not be able to be satisfied.

Embodiments of the present disclosure are directed to a system and method for resource-based scheduling of commands that consider availability of the hardware resources of the storage device for submitting the commands for processing. According to one embodiment, hardware resources of a storage device are monitored for matching/synchronizing with the software resources. Once synchronized, the software resources may be assigned to commands as desired. The assigning of the software resources to the commands may be based on, for example, a determined QoS as set forth in a service level agreement (SLA). Although QoS is used as an example of a criteria that may be used to determine allocation of the software resources, a person of skill in the art should recognize that other criteria may also be considered, such as, for example, user preferences, and/or the like.

According to one embodiment, the device driver includes a software token manager configured to identify hardware resource information, and update the software resource information accordingly. In one embodiment, the software token manager updates a number of available software resources to match a number of available hardware resources. A certain portion of the available software resources may be reserved for one or more queues of the device driver based on, for example, QoS expectations. The remaining software resources may be part of a shared pool. In one embodiment, the software token manager assigns software tokens/resources from the shared pool, to commands in the software queues, for submitting the commands to the storage device for processing. According to one embodiment, if there are no software resources in the shared pool to be assigned to a command in a queue, the software token manager may access the software tokens reserved for the queue, and assign one of the reserved tokens to the command. The reserved tokens may allow commands to be processed in a timely manner for satisfying QoS requirements.

<FIG> is a block diagram of a system for resource-based scheduling of commands according to one embodiment. The system may include a host computing device <NUM> coupled to a storage device <NUM> over a storage interface bus <NUM>. The storage device <NUM> may be a non-volatile storage device such as, for example, a solid state drive (SSD), an Ethernet SSD (eSSD), Universal Serial Bus (USB) drive, Security Digial (SD) Card, embedded Multi-Media Controller (eMMC), Universal Flash Storage (UFS), and/or the like. The storage interface bus <NUM> may be, for example, a Peripheral Component Interconnect Express (PCIe) bus, Ethernet, and CXL (Compute Express Link). In one embodiment, the host computing device <NUM> transfer and receive data to and from the storage device <NUM> over the storage interface bus <NUM>, using a storage interface protocol. The storage interface protocol may be, for example, a non-volatile memory express (NVMe) protocol or any other like protocol that uses queues for storing commands to be processed.

In one embodiment, the host computing device <NUM> includes one or more applications <NUM> running in an application layer of the host computing device <NUM>. The one or more applications <NUM> may be software applications that are stored in host memory space for execution by a processor. In one embodiment, the one or more applications <NUM> may send commands to the storage device <NUM> for processing. For example, the one or more applications may issue read commands for reading data from the storage device <NUM>, write commands for writing data into the storage device <NUM>, and/or other input/output (I/O) requests.

In one embodiment, the host computing device <NUM> includes a device driver <NUM> configured to interface with the storage device <NUM>. In one embodiment, the device driver <NUM> is implemented as software instructions that are stored in the host memory, and which are executed by the processor. The device driver <NUM> may include one or more queues 212a, 212b (hereinafter referred to as software queues <NUM>). The software queues <NUM> may include, for example, one or more submission queues and completion queues. The submission queues may be configured to store commands/requests submitted by the various applications <NUM>. The completion queues may be configured to store completion messages for the commands/requests processed by the by the storage device <NUM>.

The one or more software queues <NUM> (e.g. submission queues) may be dedicated to store certain types of commands from the host computing device <NUM>. For example, one queue may be dedicated to store read commands from the applications, while another queue may be dedicated to store write commands. In one embodiment, certain QoS requirements may be imposed on the software queues <NUM> based on one or more SLAs. For example, a certain number of the commands in the software queues <NUM> may be expected to be processed in a given time period. Different QoS requirements may be associated with the different software queues <NUM>.

The device driver <NUM> may further include a token manager <NUM> configured to manage software resources that many influence the scheduling of commands that are to be processed. The software resources may be, for example, software tokens and/or queues <NUM>. In one embodiment, the software token manager <NUM> is configured to determine availability of software tokens based on information from the storage device <NUM>, and assign available tokens to commands stored in the software queues <NUM>. The software token manager <NUM> may be configured to maintain availability information on a per controller and/or token type basis. The assigned tokens may be from a shared pool when tokens are available in the shared pool, or from a pool that is reserved for a queue, for commands submitted to the queue.

In one embodiment, the storage device <NUM> includes a host interface layer (HIL) <NUM> for interfacing between the host computing device <NUM> and a device control subsystem <NUM>. The HIL <NUM> may include, without limitation, one or more controllers <NUM> and one or more queues 220a-220c (hereinafter referred to as hardware queues <NUM>). The depth of the hardware queues <NUM> may depend, for example, on a processing power of the storage device <NUM>. In one embodiment, different queues may be maintained for different types of processing by the storage device <NUM>. For example, one hardware queue may store commands for processing by a hardware acceleration engine, and another queue may store commands for processing via firmware.

The controllers <NUM> may be implemented via one or more processors such as, for example, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). One of the one or more controllers <NUM> may be associated with one or more hardware queues <NUM>. The one of the controllers <NUM> may be configured to manage and assign hardware resources to commands submitted for processing in the software queues <NUM>. The hardware resources may be, for example, hardware tokens and/or queues <NUM>.

In one embodiment, the one or more controllers <NUM> are configured to fetch I/O requests/commands from one or more of the software queues <NUM>, and store the fetched requests into the one or more of hardware queues <NUM> corresponding to the one or more controllers <NUM>. In some embodiments, the requests may be fetched and submitted by the device driver <NUM>.

The commands stored in the hardware queues <NUM> may be assigned hardware tokens for processing. In one embodiment, different types of hardware tokens may be maintained and assigned depending on the type and number of hardware queues <NUM>. For example, a first type of hardware token may be assigned to a first hardware queue 220a dedicated for processing by a hardware acceleration engine, while a second type of hardware token may be assigned to a second hardware queue 220b dedicated for processing via firmware.

The number of available hardware tokens to be assigned to the commands in the different hardware queues <NUM> may depend on the number of commands already in the queues. In one embodiment, the tokens are assigned by the controller <NUM> based on hardware queue type. In one embodiment, hardware tokens are automatically assigned upon storing of commands in the one or more hardware queues <NUM>. Commands that have assigned hardware tokens may be scheduled for processing by the device control subsystem <NUM>.

In one embodiment, the device control subsystem <NUM> interacts with the controllers <NUM> for executing commands requested by the applications <NUM>. The device control subsystem <NUM> may include, without limitation, one or more processors <NUM> and one or more media interface(s) <NUM>. The one or more processors <NUM> may be configured to execute computer-readable instructions for processing commands to and from the controllers <NUM>, and for managing operations of the storage device <NUM>. The computer-readable instructions executed by the one or more processors <NUM> may be, for example, firmware code.

In one example, the one or more processors <NUM> may be configured to interact with the controllers <NUM> for receiving write or read commands to or from NVM media <NUM>. The one or more processors <NUM> may interact with the NVM media <NUM> over the media interface <NUM> for effectuating the write or read actions. The NVM media <NUM> may include one or more types of non-volatile memory such as, for example, flash memory.

In one embodiment, the storage device <NUM> further includes an internal memory <NUM> for short-term storage or temporary memory during operation of the storage device <NUM>. The internal memory <NUM> may include a DRAM (dynamic random access memory), SRAM (static random access memory), and/or DTCM (Data Tightly Coupled Memory). The internal memory <NUM> may be used to store, for example, the hardware queues <NUM>.

<FIG> is a block diagram of various abstraction layers of the system of <FIG> according to one embodiment. For convenience of explanation, <FIG> will be described with reference to <FIG>. The abstraction layers of the host computing device <NUM> may include an application layer <NUM> and a device driver layer <NUM>. The application layer <NUM> may be configured to generate commands based on I/O requests issued by the applications <NUM>. The commands generated by the application layer <NUM> may confirm to the storage interface protocol of the storage interface bus <NUM> used for communicating with the storage device <NUM>.

In one embodiment, the device driver layer <NUM> includes the software queues <NUM> and token manager <NUM>. The commands generated by the application layer <NUM> are stored in the queues <NUM> in the device driver <NUM> based on, for example, the command type. The software token manager <NUM> may determine availability of software tokens based on information from the storage device <NUM>, and assign available tokens to commands stored in the software queues <NUM>. In the event that all tokens are assigned, service may be rejected. In one embodiment, a command may be allowed to be queued in the software queues <NUM>, but if no available tokens exist, the command may not be submitted to the hardware queues <NUM> for processing.

In one embodiment, the storage device <NUM> includes a storage layer <NUM> that may include various sublayers. The various sublayers may include the host interface layer <NUM> as well as other layers for interfacing with the NVM media <NUM> such as, for example, a flash translation layer <NUM> and flash interface layer <NUM>. The NVM media <NUM> may be included in the storage layer <NUM> as flash media <NUM>.

<FIG> is a conceptual block diagram of <FIG> for scheduling commands in software and hardware queues according to one embodiment. The software token manager <NUM> may manage information on available software tokens <NUM>, including information on a number of available software tokens in a shared pool, and information on a number of available software tokens in a reserved pool. In one embodiment, the software token manager <NUM> maintains token availability information for the queues <NUM> managed by a controller of the various controllers <NUM> in the storage device <NUM>. In the event that the storage device <NUM> includes multiple controllers <NUM>, the software token manager <NUM> may maintain multiple sets of token availability information for the queues managed by the multiple controllers <NUM>.

The software token manager <NUM> may also maintain availability information for different types of tokens in a given set of available tokens. For example, different tokens may be assigned based on whether the command is to be processed by a hardware acceleration engine, or via firmware. The software token manager <NUM> may be configured to monitor availability of tokens for hardware acceleration processing, and tokens for firmware processing, and assign an appropriate type of token to the command based on availability.

In one embodiment, a certain number of software tokens may be reserved in the reserved pool for each software queue <NUM>. The reservation may also be based on different token types. The number of software tokens reserved for the software queues may be the same or different for the different queues and token types.

In one embodiment, the number of tokens reserved for a queue <NUM> (which should be understood to also include a token type) may be determined dynamically based on one or more criteria. The criteria may be, for example, usage statistics and/or SLA requirements. In one embodiment, the software token manager <NUM> is configured to identify the SLA requirements (e.g. required input/output per second, bandwidth, etc.) for a queue <NUM>, and dedicate a portion of available software tokens <NUM> based on the SLA requirements. For example, if the bandwidth requirements for a queue is BW1, and a total available bandwidth is BW, a number of reserved tokens may be based on a ratio of BW1 to BW (BW1/BW).

In some embodiments, the software token manager <NUM> is configured to collect historical information relating to usage of the software queues, including time periods of the usage, amount of usage, applications responsible for the usage, and the like, and dynamically adjust the reserved number of tokens for the queues based on the historical information. In this manner, when a certain time period arrives when usage of one of the queues <NUM> is predicted to be high, the software token manager <NUM> may be configured to reserve more tokens for the queue based on the prediction.

In yet some embodiments, the software token manager <NUM> receives information from the various applications <NUM> as to a number and type of commands that are expected (e.g. in the next millisecond) from the applications <NUM>. The software token manager <NUM> may dynamically adjust the number of reserved tokens for the affected queues, based on the information. For example, if a software queue is expected to be more highly utilized than a hardware queue, the number of reserved tokens for the software queue for a given time period may be proportionally higher than the number of reserved tokens for the hardware queue.

In one example, a total number of available software tokens <NUM> identified by the software token manager <NUM> for a particular controller <NUM>, at a given time window, is <NUM> tokens. The number of available software tokens <NUM> may match a total number of hardware tokens <NUM> associated with a particular controller <NUM>. The software token manager <NUM> may be configured to reserve a certain number of the available software tokens <NUM>, in a reserved pool. For example, <NUM>% of the available tokens (<NUM> tokens) may be allocated into the reserved pool, while the remaining <NUM> tokens may be allocated to the shared pool. In the event that a first one of the software queues 212a (Q1) has <NUM> commands stored in the queue, and a second one of the software queues 212b (Q2) has two commands stored in the queue, the software token manager <NUM> may be configured to schedule the commands in Q1 212a for processing based on a scheduling algorithm such as, for example, a scheduling algorithm such as round robin, weighted round robin, fixed priority, or the like. The commands in Q1 212a may be allocated the <NUM> available software tokens in the shared pool for allowing the commands to be submitted to the associated hardware queue 220a for processing. The <NUM> commands submitted to the hardware queue 220a may consume <NUM> hardware tokens from the pool of available hardware tokens <NUM>.

The two commands in Q2 212b may be selected next for processing according to the scheduling algorithm. Assuming that the <NUM> tokens assigned to the command in Q1 212a have not yet been released, there are no more tokens in the shared pool to be assigned to the commands in Q2 212b. However, given that <NUM> tokens have been allocated to the reserved pool, and divided amongst the various software queues <NUM>, two of the software tokens in the reserved pool that have been allocated to Q2 212b are assigned to the two commands in Q2 212b. In this regard, even though there are no more available software tokens in the shared pool, the two commands in Q2 212b may still be submitted to the corresponding hardware queue 220a for processing via the reserved software tokens, allowing fairness in scheduling to be maintained. The two commands submitted to the hardware queue 220a may consume two tokens from the available hardware tokens, causing a total number of available hardware tokens <NUM> to decrease to <NUM>.

In one embodiment, information on the updated hardware tokens is exposed to the software token manager <NUM> via commands or via an application programming interface (API) <NUM>. The information may then be used by the software token manager <NUM> to synchronize its software tokens <NUM>. Information on updated tokens may be pushed to the software token manager <NUM>, or fetched by the software token manager from the storage device <NUM>, on a periodic (regular or irregular) basis. For example, the controller <NUM> may submit a completion message to the device driver <NUM> using an interrupt signal, based on completing the processing of one or more commands by the processor <NUM>, which in turn may deallocate/free-up one or more corresponding hardware tokens <NUM>. The completion message may be transmitted to the software token manager <NUM>, and/or submitted to a completion queue of the one or more software queues <NUM>. When submitted to the completion queue, the software token manager <NUM> may be configured to periodically fetch completion messages from the completion queue, and update the available software tokens <NUM> accordingly. For example, when one or more completion messages associated with five processed commands are received by the software token manager <NUM>, the software token manager <NUM> may deallocate five corresponding software tokens previously allocated to the commands.

<FIG> is a flow diagram of a process for resource-based scheduling of commands stored in the software queues <NUM> according to one embodiment. It should be understood that the sequence of steps of the process is not fixed, but can be altered into any desired sequence as recognized by a person of skill in the art. For convenience of explanation, <FIG> will be described with reference to <FIG>.

The process starts, and at block <NUM>, the software token manager <NUM> receives a request for a software token for a command in one of the software queues <NUM>. In some embodiments, a placement of a command in the queue may be deemed to be a request for a software token. The request may include an identifier of the queue storing the command, and a token type.

At block <NUM>, the software token manager <NUM> determines whether information on available software tokens should be updated. Software tokens may be updated upon satisfaction of a criterion, such as, for example, expiration of a time period, or upon determination that there are no available software tokens in the shared and reserved pools when the request is received.

If the token availability information is to be updated, the software token manager <NUM> obtains information on the available hardware tokens, and synchronizes the available software tokens to match the available hardware tokens. This may include, for example, receiving an interrupt signal from the one or more controllers <NUM> for receiving the update information, or querying one or more completion queues for information on completed commands by the storage device <NUM>. In one embodiment, the update information may include token type information for allowing the software token manager <NUM> to synchronize a number of available software tokens on a per token type basis.

At block <NUM>, the software token manager <NUM> reserves a certain number of the available tokens per queue, and per token type. For example, assuming that the total number of available tokens is <NUM>, the software token manager <NUM> may allocate <NUM>% of the available tokens to a first one of the queues, and <NUM>% of the available tokens to a second one of the queues. The percentages may be static (e.g. manually set), or determined dynamically. Other percentages are also possible. The number of reserved tokens may be based, for example, on expected usage of the queue and/or queue type, QoS expectations, and/or the like.

At block <NUM>, remaining available tokens are allocated as shared tokens in a shared pool.

At block <NUM>, a determination is made as to whether shared tokens in the shared pool satisfy a condition. The condition may be, for example, availability of shared tokens to be allocated to the command. If the answer is YES, an available token in the shared pool is allocated to the command at block <NUM>. The command may now be submitted/fetched by the controller <NUM> for processing, causing consumption of a hardware token.

At block <NUM>, the software token manager <NUM> decreases the number of available software tokens in the shared pool by one.

Referring again to block <NUM>, if a determination is made that the shared tokens do not satisfy the condition (e.g. there are no shared tokens in the shared pool), the software token manager <NUM> determines, in block <NUM> whether there are any reserved tokens for the queue and token type associated with the command. If the answer is YES, an available token in the reserved pool for the queue and token type is allocated to the command at block <NUM>.

At block <NUM>, the software token manager <NUM> decreases the number of reserved tokens for the queue and token type, by one.

If there are no shared or reserved tokens to be assigned to the command, the command may remain in the queue without being serviced, until software tokens become available again.

In some embodiments, the systems and method for resource-based scheduling of command discussed above, are implemented in one or more processors. The term processor may refer to one or more processors and/or one or more processing cores. The one or more processors may be hosted in a single device or distributed over multiple devices (e.g. over a cloud system). A processor may include, for example, application specific integrated circuits (ASICs), general purpose or special purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field programmable gate arrays (FPGAs). In a processor, as used herein, each function is performed either by hardware configured, i.e., hard-wired, to perform that function, or by more general-purpose hardware, such as a CPU, configured to execute instructions stored in a non-transitory storage medium (e.g. memory). A processor may be fabricated on a single printed circuit board (PCB) or distributed over several interconnected PCBs. A processor may contain other processing circuits; for example, a processing circuit may include two processing circuits, an FPGA and a CPU, interconnected on a PCB.

Any numerical range recited herein is intended to include all sub-ranges of the same numerical precision subsumed within the recited range. For example, a range of "<NUM> to <NUM>" is intended to include all subranges between (and including) the recited minimum value of <NUM> and the recited maximum value of <NUM>, that is, having a minimum value equal to or greater than <NUM> and a maximum value equal to or less than <NUM>, such as, for example, <NUM> to <NUM>. Any maximum numerical limitation recited herein is intended to include all lower numerical limitations subsumed therein and any minimum numerical limitation recited in this specification is intended to include all higher numerical limitations subsumed therein.

Claim 1:
A method for scheduling commands for processing by a storage device (<NUM>, <NUM>, <NUM>), the method comprising:
receiving and storing a command from an application (<NUM>) in a software queue (<NUM>-<NUM>, 212a, 212b);
obtaining information on a hardware resources (<NUM>, <NUM>) managed by the storage device (<NUM>, <NUM>, <NUM>);
synchronizing a software resources (<NUM>, <NUM>) based on the information on the hardware resources (<NUM>, <NUM>);
allocating the software resources (<NUM>, <NUM>) into a shared pool and a reserved pool, wherein the shared pool is shared by the software queue (<NUM>-<NUM>, 212a, 212b) and a hardware queue (<NUM>-<NUM>, 220a-c), and the reserved pool is reserved for the software queue (<NUM>-<NUM>, 212a, 212b);
determining a condition of the software resources (<NUM>, <NUM>) in the shared pool, wherein the condition is availability of the software resources (<NUM>, <NUM>) in the shared pool;
allocating, to the command, one of the software resources (<NUM>, <NUM>) in the shared pool, based on a first determination of the condition that there are software resources in the shared pool to be assigned to the command; and
allocating, to the command, one of the software resources (<NUM>, <NUM>) in the reserved pool, based on a second determination of the condition that is different from the first determination,
wherein, based on the allocating of the one of the software resources (<NUM>, <NUM>), the command is stored in the hardware queue (<NUM>-<NUM>, 220a-c) associated with the storage device (<NUM>, <NUM>, <NUM>) for processing by the storage device (<NUM>, <NUM>, <NUM>).