Examples include selectively adjusting I/O Q-connections between an NVMe controller and a storage device in an NVMe system. In some examples, a utilization time of a host port in an NVMe controller is determined. In response to determining that the utilization time of the host port is lower than a host port utilization threshold and a number of I/O Q-connections at the storage device is less than an I/O Q-connection threshold for the storage device, a candidate list of storage devices is created, each storage devices included in the candidate list having an average service time greater than or equal to an average service time of a host port associated with the storage device. For each storage device included in the candidate list, processing time and I/O block size of I/O requests at the storage device is determined and a number of I/O Q-connections at the storage device is selectively adjusted based on the processing time and I/O block size of I/O requests at the storage device.

BACKGROUND

Non-volatile memory (NVM) is a type of computer memory that retains its contents across power cycles and is therefore capable of being used as storage. Compared to volatile memory that needs power to retain data, NVM may continue to store data even after computer power is turned off. With respect to NVM, NVM Express (NVMe) is a storage interface specification for accessing NVM.

DETAILED DESCRIPTION

NVMe is a storage interface specification for communication between hosts and storage devices (e.g., Solid State Drives (SSDs) on a PCI Express (PCIe) bus). According to the NVMe specification, a storage device may handle thousands of I/O operations in parallel. In order to provide this benefit to enterprise class data centers, NVMe may be extended over fabrics for increased scalability and shareability. In this regard, NVMe over fabrics (NVMe-oF) is a flexible transport abstraction layer that provides for a consistent definition of NVMe over a wide range of storage networking fabrics such as Ethernet and Fibre Channel. A storage device compatible with the NVMe specification and able to process commands (e.g., read commands, write commands, administrative commands, etc.) consistent with and/or provided according to the NVMe specification is referred to as an “NVMe storage device”. Example of an “NVMe storage device” may include solid-state drives (SSDs) compatible with the NVMe specification. An “NVMe storage device” may be referred to herein as an “NVMe storage device” or simply as a “storage device”. A host may be a computing device that may access data stored in, and write data to, one or more NVMe storage devices. In an example, the host may be a server providing data services to client(s) based on the data stored at one or more of the NVMe storage devices.

The NVMe specification defines both an interface (e.g., a register-level interface) and a command protocol used to communicate with the NVMe storage devices. In a system utilizing the NVMe specification, one or more NVMe storage devices (e.g., including port(s) of the NVMe storage device(s)) may be configured to communicate with a host. Communication between the host and one or more NVMe storage devices may be implemented by an NVMe controller. The NVMe controller, also referred to as a “head node”, may be a storage array controller at a front end that can manage one or more NVMe storage devices, such as SSDs, at a back end. A host may be connected to a host port on the NVMe controller, thereby associating the host port with the host. In an example, the host port may be a physical port acting as an interface between the host and the NVMe controller. The interface between the NVMe controller and the NVMe storage device may be based on several queue pairs (i.e., paired submission and completion queues) shared between the NVMe controller (e.g., including port(s) of the NVMe controller) and the NVMe storage device (e.g., including port(s) of the NVMe storage device). The queue pairs may be located either in the host memory or in the memory provided by the NVMe storage device. In an example, the NVMe specification allows up to 64K individual queue pairs per NVMe storage device, and each queue pair can have up to 64K entries. Once the queue pairs are configured, these queue pairs may be used for almost all communication between the NVMe controller and an NVMe storage device using the command protocol. Every new entry may be submitted to an NVMe storage device using a submission command via a submission queue. When the submission command is processed, an entry (that has been previously associated with the submission queue from which the command was retrieved) may be put on a completion queue using a completion command, and an interrupt may be generated. There may be separate queue pairs for administration operations (e.g., creating and deleting queues or updating firmware on the device) and for I/O operations (e.g., Read and Write). Separate queue pairs may avoid excessive delay of I/O operations due to long-running administration operations. Each queue pair for I/O operations between the NVMe controller and an NVMe storage device may be referred to as an “I/O Q-connection” at the NVMe storage device.

Generally, NVMe storage devices can process I/O operations at a faster rate as compared to the NVMe controller. However, since a single NVMe controller may manage multiple NVMe storage devices at the back-end, the processing load at the NVMe controller may increase manifold with increase in processing load in one or more of the multiple NVMe storage devices. Thus, the NVMe controller may not be able to process commands to and from the multiple NVMe storage devices at an optimal rate and consequently Input Output Operations per Second (IOPS) between the NVMe controller at the front end and the NVMe storage devices at the back end may be reduced thereby adversely affecting performance. Therefore, the NVMe controller may become a bottleneck in achieving high IOPS between the NVMe controller and the NVMe storage devices. Further, in some cases, the number of I/O Q-connections between the NVMe controller and NVMe storage devices may not be enough to process incoming I/O load from the hosts which may lead to choking of I/O operations at the NVMe storage devices. Choking of I/O operations at the NVMe storage devices may lead to an increase in latency at the NVMe storage devices and timeouts in application(s) running in the hosts.

Examples described herein provide techniques to dynamically scale up or scale down I/O Q-connections between the NVMe controller and the NVMe storage devices based on the I/O workload, consequently improving IOPS for storage applications. The techniques may determine a utilization time of a host port in the NVMe controller. The host port may be associated with a host and may communicate with an NVMe storage device. The utilization time of the host port is indicative of a time period for which the host port is occupied with processing of I/O operations. In response to determining that the utilization time of the host port is below a host port utilization threshold and a number of I/O Q-connections at the NVMe storage device is less than an I/O Q-connection threshold for the NVMe storage device, the NVMe controller may create a candidate list of NVMe storage devices. The I/O Q-connection threshold is indicative of the maximum number of I/O Q-connections that can be simultaneously supported by an NVMe storage device for servicing I/O requests. The candidate list may include NVMe storage devices for which selective adjustment of I/O Q-connections could be considered. Each NVMe storage device included in the candidate list has an average service time greater than or equal to an average service time of a host port associated with the NVMe storage device. The host port utilization threshold is indicative of a maximum amount of the time that the host port may be occupied with processing of I/O operations. The average service time of an NVMe storage device refers to the average time taken by an NVMe storage device to process I/O operations (e.g. read or write). A host or NVMe controller may send I/O requests including commands/instructions for executing I/O operations (e.g. read or write) at NVMe storage device(s). The average service time of a host port of the NVMe controller refers to the average time taken by the host port to process an I/O operation. For each NVMe storage device included in the candidate list, the NVMe controller may determine processing time and I/O block size of I/O requests at the NVMe storage device and selectively adjust, based on the processing time and I/O block size of I/O requests, the number of I/O Q-connections at the NVMe storage device. In this manner, the number of I/O Q-connections between the NVMe controller and the NVMe storage devices may be increased or decreased, based on the workload, consequently achieving higher IOPS and improving performance. Increasing or decreasing the number of I/O Q-connections based on the workload may facilitate reducing the latency in processing I/O operations from the hosts via the storage devices and thereby reduce timeouts in applications running in the hosts.

FIG.1illustrates an example system100including an NVMe controller110(hereinafter also referred to as “controller110”) that may facilitate connecting hosts102to communicate with NVMe storage devices104(hereinafter also referred to as “storage devices104”). In an example, a storage array may include the storage devices104and may include controller110as the storage array controller of the storage array. The system100illustrated inFIG.1may include a plurality of NVMe storage devices104(designated as NVMe storage devices104-1through104-P) and a plurality of hosts102(designated as hosts102-1through102-N). Each of the NVMe storage devices104may be accessed by a corresponding subset of the hosts102. For example, a first subset of the hosts102can communicate with an NVMe storage device104-1, a second subset of the hosts102can communicate with an NVMe storage device104-2, and so on. In some examples, a given host of the hosts102can communicate with two or more of NVMe storage devices104(i.e., the given host may belong to two or more subsets of the hosts).

In an example, the controller110may be attached to, be aware of, be part of, be associated with, and/or be otherwise related to a fabric (e.g., NVMe fabrics) to which the hosts102and NVMe storage devices104are communicatively connected. The controller110may include at least one processing resource112communicatively coupled to a machine-readable storage medium114including at least analysis instructions118and I/O Q-connection control instructions120that, when executed by the at least one processing resource112, cause the controller110to perform actions described herein in relation to controller110. In some examples, the instructions of controller110may be executed in a switch (e.g., embedded in a container), in a virtual machine (VM), or in an NVMe storage device (e.g., the NVMe storage device104-1).

The controller110may facilitate connecting the hosts102to NVMe storage devices104. The hosts102can communicate to the NVMe storage device(s) based on a mapping. For example, inFIG.1, the mapping may indicate that the hosts designated102-1, . . . ,102-N can communicate with the NVMe storage devices104.

The controller110includes host ports106-1, . . . ,106-N, also referred to as host ports106. Each of the hosts102may connect with a host port, form the host ports106, thereby associating each of the host ports106with a particular host. Each of the hosts102associated with a host port, from the host ports106, may be enabled to communicate with an NVMe storage device from a plurality of NVMe storage devices104.

The controller110may include analysis instructions118and I/O Q-connection control instructions120to perform one or more functionalities of the controller110as described herein. In other examples, functionalities described herein in relation to controller110may be implemented via any combination of hardware and programming to perform the functionalities described herein in relation to controller110. The combination of hardware and programming may be implemented in a number of different ways. For example, the programming may include processor executable instructions stored on a non-transitory machine-readable storage medium and the hardware may include at least one processing resource (e.g., at least one processor, CPU, circuitry, etc.) to execute those instructions. In examples described herein, a single computing device (e.g., a storage array) may include machine-readable storage medium storing the instructions (or other programming) and the processing resource (or other hardware) to execute the instructions (or other programming), or the machine-readable storage medium storing instruction (or other programming) may be separate from and accessible by the computing device and the processing resource. In some examples, the hardware and programming may be implemented in circuitry.

In an example, a sampling interval may be configured for the controller110. The sampling interval is indicative of a time interval at which the controller110is to perform one or more functionalities for managing a number of I/O Q-connections between the controller110and the NVMe storage devices104-1to104-P. The sampling interval may be predefined based on a user input. The sampling interval may be, for example, 3600 seconds. In an example, the I/O Q-connection control instructions120may create a pair of I/O Q-connections between the controller110and each NVMe storage device104-1, . . . ,104-P. One of the pair of I/O Q-connections (i.e., a set of submission and completion queues) may be dedicated for read operations and the other of the pair of I/O Q-connections (another set of submission and completion queues) may be dedicated for write operations.

The analysis instructions118may determine a utilization time of each of the plurality of host ports106-1to106-N. Although, in the description hereinafter, the operations/functionalities are described with reference to the host port106-1and storage device104-1, similar operations/functionalities may also be performed in respect of each of the other host ports106-2to106-N and each of the storage devices104. The analysis instructions118may determine a throughput of the host port106-1based on a number of I/O request completions at the host port106-1over the sampling interval. The I/O request completions at the host port106-1may refer to I/O requests serviced/processed at the host port106-1during the sampling interval. In an example, the throughput of the host port106-1is a ratio of a number of I/O request completions at the host port106-1to the sampling interval.

Further, the analysis instructions118may determine an average service time of the host port106-1. The average service time is indicative of the average time taken for processing an I/O operation (read or write) by the host port106-1. The average service time of the host port106-1may be computed as a ratio of a busy time period of the host port106-1and the number of I/O request completions at the host port106-1over the sampling interval. The busy time period of the host port106-1refers to a time duration for which the host port106-1remains unavailable for processing/receiving I/O requests from the hosts102, such as the host102-1. The utilization time of the host port106-1may be computed as a product of the throughput of the host port106-1and the average service time of the host port106-1. The utilization time of each of the host ports106-2to106-N may also be determined in a similar manner.

The analysis instructions118may compare the utilization time of the host port106-1with a host port utilization threshold. In an example, the host port utilization threshold may be expressed in terms of percentage of the sampling interval for which the host port106-1is utilized, say 98% of the sampling interval. Further, the analysis instructions118may also check whether a number of I/O Q-connections at an NVMe storage device (such as storage device104-1) is less than an I/O Q-connection threshold for the storage device104-1. In an example, the analysis instructions118may receive, from the storage device104-1, information including a quantity of the I/O Q-connections (i.e., a number of I/O Q-connections) existing at the storage device104-1, at a point in time. In some examples, the storage device104-1may send a registration command, to the controller110, to register the number of existing I/O Q-connections at the storage device104-1with the controller110. The registration command may comprise information including the number of existing I/O Q-connections and device parameter(s) of the storage device104-1. The analysis instructions118may obtain the information including the number of existing I/O Q-connections from the registration command. The device parameter(s) of the storage device104-1may include information related to the I/O Q-connection threshold for the storage device104-1. The I/O Q-connection threshold indicates the maximum number of I/O Q-connections that can be simultaneously supported by the storage device104-1for servicing I/O requests. In an example, the I/O Q-connection threshold for the storage device104-1may depend on device driver configuration of the storage device104-1. The analysis instructions118may obtain the information including the I/O Q-connection threshold from the device parameter(s) included in the registration command. The analysis instructions118may compare the number of existing I/O Q-connections at the storage device104-1with the I/O Q-connection threshold.

In response to determining that the utilization time of the host port106-1is lower than the host port utilization threshold and the number of I/O Q-connections at the storage device104-1is less than the I/O Q-connection threshold for the storage device104-1, the analysis instructions118may create a candidate list of storage devices. The candidate list may include NVMe storage devices for which selective adjustment of I/O Q-connections could be considered. The storage device104-1may be included in the candidate list, if an average service time of the storage device104-1being greater than or equal to an average service time of the host port106-1. If the average service time of the storage device104-1is less than the average service time of the host port106-1, the storage device104-1is not added to the candidate list and other storage devices104may be considered for addition to the candidate list. Thus, each NVMe storage device included in the candidate list has an average service time greater than or equal to an average service time of a host port associated with the NVMe storage device. A host port associated with an NVMe storage device may refer to a host port which can access the resources of the NVMe storage device in an NVMe system, such as the system100.

Although, the techniques hereinafter are described with reference to controlling I/O Q-connections between the controller110and the storage device104-1, similar techniques may be employed to control I/O Q-connections between the controller110and each of the storage devices104. In an example, the analysis instructions118may determine a number of I/O request completions at the storage device104-1over the sampling interval. The I/O request completions at the storage device104-1refers to I/O requests serviced/processed at the storage device104-1during the sampling interval. The analysis instructions118may also determine a busy time period of the storage device104-1. The busy time period may indicate a time period for which the storage device104-1may remain unavailable to process new I/O requests during the sampling interval. The analysis instructions118may determine an average service time of the storage device104-1as a ratio of the busy time period of the storage device104-1to the number of I/O request completions at the storage device104-1.

In response to determining that the average service time of the storage device104-1is greater than or equal to the average service time of the host port106-1, the analysis instructions118may group the storage device104-1within the candidate list. Thus, for each of the storage devices104-1to104-P, if average service time of the storage device, is greater than or equal to the average service time of the host port106-1, they may be added to the candidate list. In an example, consider that storage devices104-1and104-2are included in the candidate list.

The analysis instructions118may also determine a visit ratio of the storage device104-1. The visit ratio of the storage device104-1is defined as the number of I/O request completions at the storage device104-1per unit time of the sampling interval. Further, a throughput of the storage device104-1may be determined based on the visit ratio. In an example, the throughput of the storage device104-1may be computed as a product of the visit ratio of the storage device104-1and the throughput of the host port106-1. The analysis instructions118may determine a service demand for the storage device104-1as a product of the visit ratio of the storage device104-1and the average service time of the storage device104-1. Further, the analysis instructions118may determine a utilization time of the storage device104-1as a product of the throughput of the host port106-1and the service demand of the storage device104-1. Likewise, a utilization time of the storage device104-2(included in the candidate list) may also be determined.

In an example, the I/O Q-connection control instructions120may determine an average time for processing read requests by the storage device104-1in the sampling interval. A read request refers to an I/O request from a host to read data from one of the storage devices. The I/O Q-connection control instructions120may check whether there is a change in the average time for processing read requests by the storage device104-1over two successive sampling intervals. In an example, the average time for processing a first set of read requests by the storage device104-1during a first sampling interval may be compared with the average time for processing a second set of read requests by the storage device104-1in a second sampling interval, where the first and second sampling intervals are successive sampling intervals. Based on the comparison, the I/O Q-connection control instructions120may determine the change in the average time for processing the read requests. In an example, the change may indicate an increase or decrease in the average time for processing the read requests by the storage device104-1. Further, the I/O Q-connection control instructions120may also check whether the utilization time of the storage device104-1is less that a storage utilization threshold, say, 95% of the sampling interval. The storage utilization threshold is indicative of a maximum amount of time for which the storage device104-1may be occupied with processing of I/O operations.

In response to determining that there is an increase in the average time for processing read requests by the storage device104-1and the storage device104-1has a utilization time less than the storage utilization threshold, the I/O Q-connection control instructions120may determine whether a number of read requests of small I/O block size received per second during the sampling interval is greater than or equal to a read I/O threshold. In an example, read requests having an I/O block size less than or equal to 4 Kilo Bytes (KB) may be referred to as read requests of small I/O block size. The read I/O threshold may indicate a certain amount, for example, 80%, of the read requests received per second by the storage device104-1during the sampling interval. Thus, the I/O Q-connection control instructions120may check whether the number of read requests of small I/O block size received per second during the sampling interval is greater than or equal to 80% of the read requests received per second by the storage device104-1.

In response to determining that the number of read requests of small I/O block size received per second during the sampling interval is greater than or equal to the read I/O threshold, the I/O Q-connection control instructions120may add an I/O Q-connection with high priority associated with read requests of small I/O block size. The I/O Q-connection may include a submission queue and a completion queue between the controller110and the storage device104-1. The I/O Q-connection is dedicated to read requests of small I/O block size received at the storage device104-1. In an example, the added I/O Q-connection may have a priority higher than an I/O Q-connection that is currently servicing the read requests at the storage device104-1. Thus, the I/O requests serviced through the added I/O ( ) connection may be processed at a higher speed.

In an example, the I/O Q-connection control instructions120may determine that the number of read requests of large I/O block size, such as I/O blocks of 512 KB, received per second during the sampling interval is greater than or equal to the read I/O threshold. In response to determining that the number of read requests of large I/O block size received per second during the sampling interval is greater than or equal to the read I/O threshold, the I/O Q-connection control instructions120may add an I/O Q-connection with normal priority associated with read requests of large I/O block size. In an example, the added I/O Q-connection may have a priority same as that of an I/O Q-connection that is currently servicing the read requests at the storage device104-1. The added I/O Q-connection is dedicated to read requests of large I/O block size received by the storage device104-1.

In response to determining that the average time for processing read requests by the storage device104-1has reduced over two successive sampling intervals, or that the storage device104-1has the utilization time greater than the storage utilization threshold, the I/O Q-connection control instructions120may check whether more than one I/O Q-connection is present between the storage device104-1and the controller110. Further, the I/O Q-connection control instructions120may determine whether there is a change in the service demand of the storage device104-1over two successive sampling intervals. In an example, the service demand of the storage device104-1in a first sampling interval may be compared with the service demand of the storage device104-1in a second sampling interval to determine the change in service demand. The first and second sampling intervals may be two successive sampling intervals. Responsive to determining that there is a decrease in the service demand of the storage device104-1over two successive sampling intervals and more than one I/O Q-connection is present between the storage device104-1and the controller110, the I/O Q-connection control instructions120may delete an I/O Q-connection between the storage device104-1and the controller110.

In an example, the I/O Q-connection control instructions120may determine an average time for processing write requests by the storage device104-1during the sampling interval. A write request refers to a request from a host to write data onto one of the storage devices104. The I/O Q-connection control instructions120may check whether there is a change in the average time for processing write requests over two successive sampling intervals. In an example, an average time for processing a first set of write requests by the storage device104-1in a first sampling interval may be compared with an average time for processing a second set of write requests in a second sampling interval to determine the change in the average time for processing write requests. In an example, the first sampling interval and second sampling interval may be successive sampling intervals. In an example, the change may indicate an increase or decrease in the average time for processing write requests by the storage device104-1. Further, the I/O Q-connection control instructions120also checks whether the utilization time of the storage device104-1is less that the storage utilization threshold.

In response to determining that there is an increase in the average time for processing the write requests by the storage device104-1and the storage device104-1has a utilization time less than the storage utilization threshold, the I/O Q-connection control instructions120may determine whether a number of write requests of small I/O block size, such as I/O block size less than or equal to 4 KB, received per second during the sampling interval is greater than or equal to a write I/O threshold. The write I/O threshold may indicate a certain amount, for example, 80%, of the write requests received per second by the storage device104-1during the sampling interval. Thus, the I/O Q-connection control instructions120may check whether at least 80% of the write requests received per second by the storage device104-1during the sampling interval is of small I/O block size.

In response to determining that the number of write requests of small I/O block size received per second during the sampling interval is greater than or equal to the write I/O threshold, the I/O Q-connection control instructions120may add an I/O Q-connection associated with write requests. The added I/O Q-connection is dedicated to process write requests at the storage device104-1. In an example, the I/O Q-connection control instructions120may determine that a number of write requests of large I/O block size, such as I/O blocks of 512 KB, received per second during the sampling interval is greater than or equal to the write I/O threshold. In response to determining that the number of write requests received per second during the sampling interval is greater than or equal to the write I/O threshold, the I/O Q-connection control instructions120may add a completion queue associated with write requests. The completion queue associated with write requests may refer to a completion queue between the controller110and the storage device104-1dedicated to processing of write requests at the storage device104-1. In the manner, as described above, the I/O Q-connection control instructions120may selectively adjust the number of I/O Q-connections at the storage device104-1based on processing time and I/O block size of I/O requests at the storage device104-1. Likewise, in a similar manner, the I/O Q-connection control instructions120may selectively adjust the number of I/O Q-connections at the storage device104-2which is considered to be included in the candidate list. Thus, for each NVMe storage device included in the candidate list, processing time and I/O block size of I/O requests at the NVMe storage device are determined based on which the number of I/O Q-connections at the NVMe storage device is selectively adjusted.

FIG.2is a block diagram of a computing system200including a processing resource202and a machine-readable storage medium204encoded with example instructions206,208,210, and212to control I/O Q-connections between an NVMe controller (such as the NVMe controller110ofFIG.1) and a storage device (such as the storage device104-1ofFIG.1), in accordance with an example.

In some examples, the machine-readable storage medium204may be accessed by the processing resource202. The processing resource202may execute instructions (i.e., programming or software code) stored on the machine-readable storage medium204. The instructions206,208,210, and212ofFIG.2, when executed by the processing resource202, may implement various aspects of selectively adjusting I/O Q-connections between the NVMe controller and the storage device. In an example, the instructions206and208may be included within the analysis instructions118and the instructions210and212may be included within the I/O Q-connection control instructions120ofFIG.1. In some examples, the computing system200may be included in (e.g., as part of) an NVMe controller (e.g., the NVMe controller110ofFIG.1). For ease of illustration,FIG.2will be described with reference toFIG.1. In certain examples, the instructions206-212may be executed for performing the functionalities of the NVMe controller110and one or more methods, such as, the methods300and400described inFIGS.3and4A-4D. In certain examples, as an alternative or in addition to executing the instructions206-212, the processing resource202may include at least one IC, other control logic, other electronic circuitry, or combinations thereof that include a number of electronic components for performing the functionalities described herein as being performed by the NVMe controller110.

Instructions206, when executed by the processing resource202, may determine a utilization time of the host port106-1in the NVMe controller110. The host port106-1is associated with a host102-1and is to communicate with an NVMe storage device104-1.

In response to determining that the utilization time of the host port106-1is lower than a host port utilization threshold and a number of I/O Q-connections at the storage device104-1is less than the I/O Q-connection threshold for the storage device104-1, instructions208may create a candidate list of NVMe storage devices. Each NVMe storage device (such as the NVMe storage device104-1) included in the candidate list has an average service time greater than or equal to an average service time of a host port associated with the NVMe storage device (such as the host port106-1associated with the NVMe storage device104-1).

For the NVMe storage device104-1included in the candidate list, instructions210, when executed by the processing resource202, may determine processing time and I/O block size of I/O requests at the NVMe storage device104-1. . . Instructions212, when executed by the processing resource202, may selectively adjust, based on the processing time and I/O block size of I/O requests, the number of I/O Q-connections at the NVMe storage device104-1.

The instructions206-212may include various instructions to execute at least a part of the methods described inFIGS.3-4A-4D(described later). Also, although not shown inFIG.2, the machine-readable storage medium204may also include additional program instructions to perform various other method blocks described inFIGS.3and4A-4D.

FIGS.3and4A-4Ddepict flowcharts of example methods300and400for controlling I/O Q-connections between NVMe storage devices (e.g., the NVMe storage devices104ofFIG.1) and an NVMe controller (NVMe controller110ofFIG.1). For ease of illustration, the execution of example methods300and400is described in details below with reference toFIG.1. Although the below description is described with reference to the NVMe controller110ofFIG.1, however other applications or devices suitable for the execution of methods300and400may be utilized. Furthermore, although the below description is described with reference to the NVMe storage device104-1ofFIG.1, however the methods300and400are applicable to other NVMe storage devices. In some examples, the methods300and400may, individually, be executed for each NVMe storage device present in the system100. The method steps at various blocks depicted inFIGS.3and4A-4Dmay be performed by the NVMe controller110. In some examples, the methods300and400, individually, at each such method blocks may be executed by the computing system200via the processing resource202that executes the instructions206-212stored in the non-transitory machine-readable storage medium204. Additionally, implementation of methods300and400is not limited to such examples. Although the flowcharts ofFIGS.3and4A-4D, individually, show a specific order of performance of certain functionalities, methods300and400are not limited to such order. For example, the functionalities shown in succession in the flowcharts may be performed in a different order, may be executed concurrently or with partial concurrence, or a combination thereof.

InFIG.3, at block302, the method300may include determining a utilization time of the host port106-1in the NVMe controller110. The host port106-1is associated with a host102-1and is to communicate with an NVMe storage device104-1. In an example, a throughput of the host port106-1may be determined based on a number of I/O request completions at the host port106-1over the sampling interval. In an example, the throughput of the host port106-2is a ratio of a number of I/O request completions at the host port106-1to the sampling interval. Further, an average service time of the host port106-1may be determined. The average service time is indicative of the average time taken for servicing an I/O request at the host port106-1. The average service time of the host port106-1may be computed as a ratio of a busy time period of the host port106-1and the number of I/O request completions at the host port106-1over the sampling interval. The busy time period of the host port106-1refers to a time duration for which the host port106-1remains unavailable for processing/receiving I/O requests from the host102-1. The utilization time of the host port106-1may be computed as a product of the throughput of the host port106-1and the average service time of the host port106-1.

In response to determining that the utilization time of the host port106-1is lower than a host port utilization threshold and the number of I/O Q-connections at the storage device104-1is less than the I/O Q-connection threshold for the storage device104-1, at block304, the method300may include creating a candidate list of NVMe storage devices. Each NVMe storage device, such as the NVMe storage device104-1, included in the candidate list, has an average service time greater than or equal to an average service time of the host port106-1. The candidate list may include NVMe storage devices for which selective adjustment of I/O Q-connections could be considered.

At block306, the method300may include, for each storage device, such as the storage device104-1, included in the candidate list, determining a processing time and I/O block size of I/O requests at the storage device104-1. At block308, the method300may include, for the storage device104-1included in the candidate list, selectively adjusting, based on the processing time and I/O block size of I/O requests, the number of I/O Q-connections at the storage device104-1.

Turning now toFIG.4A, at block402, the method400may include configuring a sampling interval based on a user input. At block404, the method400may include determining a number of host ports106-1to106-N in an NVMe controller110. At block406, the method400may include determining a number of NVMe storage devices104-1to104-P associated with the NVMe controller110. In an example, the storage devices104may register with the NVMe controller110using a registration request and thereby associate with the NVMe controller110.

At block408, the method400may include creating a pair of I/O Q-connections between the NVMe controller110and an NVMe storage device, such as the NVMe storage device104-1. Each of the pair of I/O Q-connections may include a submission queue for read and write operations and a completion queue. At block410, the method400may include determining, based on I/O request completions over the sampling interval, a throughput of a host port106-1. In an example, the throughput of the host port106-2is a ratio of a number of I/O request completions at the host port106-1to the sampling interval.

At block412, the method400may include determining, based on a busy time period of the host port106-1and the I/O request completions, an average service time of the host port106-1. In an example, the average service time of the host port106-1may be computed as a ratio of a busy time period of the host port106-1and the number of I/O request completions at the host port106-1over the sampling interval. The busy time period of the host port106-1refers to a time duration for which the host port106-1remains unavailable for processing/receiving I/O requests from the host102-1. At block414, the method400may include computing a utilization time of the host port106-1, abbreviated as T(U)HOST PORT, as a product of the throughput of the host port106-1and the average service time of the host port106-1.

At block416, the method400may include comparing T(U)HOST PORTwith a host port utilization threshold, abbreviated as T(U)HOST PORTTHRESHOLD. Further, the method400may include checking whether the number of I/O Q-connections at the storage device104-1is less than an I/O Q-connection threshold for the storage device104-1. The I/O Q-connection threshold indicates the maximum number of I/O Q-connections that can be simultaneously supported by the storage device104-1for servicing I/O requests. In an example, the I/O Q-connection threshold for the storage device104-1may depend on device driver configuration of the storage device104-1. The information including the I/O Q-connection threshold may be obtained from the device parameter(s) included in a registration command received by the controller110from the storage device104-1.

In response to determining that T(U)HOST PORTfor the host port106-2is almost equal to or greater than the T(U)HOST PORTTHRESHOLD or that the number of I/O Q-connections at the storage device104-1is greater than or equal to the I/O Q-connection threshold for the storage device104-1, the steps410to416of the method400may be carried out for another host port say host port106-2.

In response to determining that the T(U)HOST PORTfor the host port106-1is lower than the T(U)HOST PORTTHRESHOLD and the number of I/O Q-connections at the storage device104-1is less than the I/O Q-connection threshold for the storage device104-1, the method400may include creating a candidate list of NVMe storage devices corresponding to the host port106-1. The candidate list may include NVMe storage devices for which selective adjustment of I/O Q-connections could be considered. At block418, the method400may include, determining a number of I/O request completions at the NVMe storage device104-1. At block420, the method400may include determining a visit ratio of the NVMe storage device104-1. The visit ratio of the storage device104-1is defined as the number of I/O request completions by the storage device104-1per unit time of the sampling interval.

At block422ofFIG.4B, the method400may include determining a throughput of the NVMe storage device104-1based on the visit ratio. In an example, the throughput of the storage device104-1may be computed as a product of the visit ratio of the storage device104-1and the throughput of the host port106-1. At block424, the method400may include determining, based on a busy time period of the NVMe storage device104-1and the number of I/O request completions, an average service time of the NVMe storage device104-1. The busy time period of the storage device104-1may indicate a time period for which the storage device104-1may remain unavailable to process new I/O requests during the sampling interval.

At block426, the method400may include comparing the average service time of the NVMe storage device104-1with the average service time of the host port106-1. In response to determining that the average service time of the NVMe storage device104-1is greater than or equal to the average service time of the host port106-1, at block430, the method may include grouping the NVMe storage device within the candidate list of NVMe storage devices. In response to determining that the average service time of the NVMe storage device104-1is less than the average service time of the host port106-1, the steps418to426may be carried for NVMe storage device104-2to check whether the NVMe storage device104-2is to be grouped within the candidate list. Thus, for each of the NVMe storage devices104, the steps418to426may be carried out to check whether the NVMe storage device104-2is to be grouped within the candidate list. Consider that the NVMe storage device104-1is included in the candidate list.

At block432, the method400may include determining a service demand for the NVMe storage device104-1. The service demand for the storage device104-1may be determined as a product of the visit ratio of the storage device104-1and the average service time of the storage device104-1. At block434, the method400may include determining a utilization time of the NVMe storage device104-1(T(U)STORAGE) based on the service demand for the storage device104-1and throughput of the host port106-1. T(U)STORAGEmay determined as a product of service demand for the storage device104-1and throughput of the host port106-1.

At block436, the method400may include comparing the average time for processing a first set of read requests by the storage device104-1during the current sampling interval (Current T(R)AVG) with the average time for processing a second set of read requests by the storage device104-1in a previous sampling interval (Previous T(R)AVG). Further, at block436, the method400may further include checking whether a utilization time of the NVMe storage device104-1(T(U)STORAGE) is less that a storage utilization threshold (T(U)STORAGETHRESHOLD), say, 95% of the sampling interval.

In response to determining that there is an increase in the average time for processing read requests by the storage device104-1and the storage device104-1has the utilization time (T(U)STORAGE) less than the storage utilization threshold (T(U)STORAGETHRESHOLD) (YES' branch from block436), at block438ofFIG.4C, the method400may include checking whether a number of read requests of small I/O block size received, at the storage device104-1, per second during the sampling interval is greater than or equal to a read I/O threshold. The read I/O thresholds may be 80% of the read requests received per second by the storage device104-1during the sampling interval.

In response to determining that the number of read requests of small I/O block size received per second during the sampling interval is greater than or equal to the read I/O threshold (YES' branch from block438), at block440, the method400may include adding an I/O Q-connection with high priority associated with read requests of small I/O block size received at the NVMe storage device104-1. The I/O Q-connection may include a submission queue and a completion queue between the NVMe controller110and the storage device104-1. The I/O Q-connection is dedicated to read requests having small I/O block sizes. In response to determining that a number of read requests of large I/O block size, such as I/O blocks of 512 KB, received per second during the sampling interval is greater than or equal to the read I/O threshold (‘NO’ branch from block438), at block442, the method400may include adding an I/O Q-connection with normal priority associated with read requests of large I/O block size received at the NVMe storage device104-1. The added I/O Q-connection is dedicated to read requests of large I/O block size received by the storage device104-1.

In response to determining that there is a decrease in the average time for processing read requests by the storage device104-1or the storage device104-1has the utilization time (T(U)STORAGE) almost equal to or greater than the storage utilization threshold (T(U)STORAGETHRESHOLD) (‘NO’ branch from block436), at block444ofFIG.4C, the method400may include checking whether more than one I/O Q-connection is present between the storage device104-1and the NVMe controller110. Further, at block444, the I/O Q-connection control instructions120may check whether there is a change in the service demand of the storage device104-1over two successive sampling intervals. In an example, the service demand of the storage device104-1in a first sampling interval may be compared with the service demand of the storage device104-1in a second sampling interval to determine the change in service demand. The first and second sampling intervals may be two successive sampling intervals. Responsive to determining that there is a decrease in the service demand of the storage device104-1over two successive sampling intervals and more than one I/O Q-connection is present between the storage device104-1and the NVMe controller110(‘YES’ branch from block444), at block446, the method400may include deleting an I/O Q-connection, associated with read I/O, between the storage device104-1and the NVMe controller110.

In response to determining that there is an increase in the service demand of the storage device104-1over two successive sampling intervals or a single I/O Q-connection is present between the storage device104-1and the NVMe controller110(‘NO’ branch from block444), at block448, the method400may include comparing the average time for processing a first set of write requests by the storage device104-1during the current sampling interval (Current T(W)AVG) with the average time for processing a second set of read requests by the storage device104-1in a previous sampling interval (Previous T(W)AVG). Further, at block448, the method400may further include checking whether a utilization time of the NVMe storage device104-1(T(U)STORAGE) is less that a storage utilization threshold (T(U)STORAGETHRESHOLD), say, 95% of the sampling interval.

In response to determining that there is an increase in the average time for processing write requests by the storage device104-1(i.e., Current T(w)avg>previous T(w)avg) and the storage device104-1has the utilization time (T(U)STORAGE) less than the storage utilization threshold (T(U)STORAGETHRESHOLD) (YES' branch from block448), at block450ofFIG.4D, the method400may include determining whether a number of write requests of small I/O block size, such as I/O block size less than or equal to 4 KB received per second during the sampling interval is greater than or equal to a write I/O threshold. The write I/O threshold may be 80% of the write requests received per second by the storage device104-1during the sampling interval.

In response to determining that the number of write requests of small I/O block size received per second during the sampling interval is greater than or equal to the write I/O threshold (YES' branch from block450), at block452, the method400may include adding an I/O Q-connection associated with write requests received at the NVMe storage device104-1. The I/O Q-connection may include a submission queue and a completion queue between the NVMe controller110and the storage device104-1. In response to determining that a number of write requests of large I/O block size, such as I/O blocks of 512 KB, received per second during the sampling interval is greater than or equal to the write I/O threshold (‘NO’ branch from block450), at block454, the method400may include adding a completion queue associated with write requests received at the NVMe storage device104-1.

In response to determining that there is a decrease in the average time for processing write requests by the storage device104-1(i.e., Current T(w)avg<previous T(w)avg) or the storage device104-1has the utilization time (T(U)STORAGE) almost equal to or greater than the storage utilization threshold (T(U)STORAGETHRESHOLD) (NO′ branch from block448), at block456ofFIG.4D, the method400may include checking whether more than one I/O Q-connection associated with write I/O is present between the storage device104-1and the NVMe controller110. Further, at block456, the method400may include checking whether there is a change in the service demand of the storage device104-1over two successive sampling intervals. In an example, the service demand of the storage device104-1in a first sampling interval may be compared with the service demand of the storage device104-1in a second sampling interval to determine the change in service demand. The first and second sampling intervals may be two successive sampling intervals. Responsive to determining that there is a decrease in the service demand of the storage device104-1over two successive sampling intervals and more than one I/O Q-connection for write requests is present between the storage device104-1and the NVMe controller110(‘YES’ branch from block456), at block458, the method400may include deleting an I/O Q-connection, associated with write I/O, between the storage device104-1and the NVMe controller110.

Examples are described herein with reference toFIGS.1-4D. It should be noted that the description and figures merely illustrate the principles of the present subject matter along with examples described herein, and should not be construed as limiting the present subject matter. Although some examples may be described herein with reference to a single NVMe storage device, examples may be utilized for several NVMe storage devices. Furthermore, any functionality described herein as performed by a component (e.g., an NVMe controller, an NVMe storage device or a host) of a system may be performed by at least one processing resource of the component executing instructions (stored on a machine-readable storage medium) to perform the functionalities described herein. Various implementations of the present subject matter have been described below by referring to several examples.

In examples described herein, functionalities described as being performed by “instructions” may be understood as functionalities that may be performed by those instructions when executed by a processing resource. In other examples, functionalities described in relation to instructions may be implemented by any combination of hardware and programming.

As used herein, a “computing device” may be a server, storage device, storage array, desktop or laptop computer, switch, router, or any other processing device or equipment including a processing resource. In examples described herein, a processing resource may include, for example, one processor or multiple processors included in a single computing device or distributed across multiple computing devices. As used herein, a “processor” may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to retrieve and execute instructions, other electronic circuitry suitable for the retrieval and execution instructions stored on a machine-readable storage medium, or a combination thereof. In examples described herein, a processing resource may fetch, decode, and execute instructions stored on a storage medium to perform the functionalities described in relation to the instructions stored on the storage medium. In other examples, the functionalities described in relation to any instructions described herein may be implemented in the form of electronic circuitry, in the form of executable instructions encoded on a machine-readable storage medium, or a combination thereof. The storage medium may be located either in the computing device executing the machine-readable instructions, or remote from but accessible to the computing device (e.g., via a computer network) for execution. In the examples illustrated inFIGS.1and4A-4D, NVMe controller110may be implemented by one machine-readable storage medium, or multiple machine-readable storage media.

As used herein, a “machine-readable storage medium” may be any electronic, magnetic, optical, or other physical storage apparatus to contain or store information such as executable instructions, data, and the like. For example, any machine-readable storage medium described herein may be any of RAM, EEPROM, volatile memory, non-volatile memory, flash memory, a storage drive (e.g., an HDD, an SSD), any type of storage disc (e.g., a compact disc, a DVD, etc.), or the like, or a combination thereof. Further, any machine-readable storage medium described herein may be non-transitory. In examples described herein, a machine-readable storage medium or media may be part of an article (or article of manufacture). All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and/or all of the elements of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and/or elements are mutually exclusive.

The foregoing description of various examples has been presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive or limiting to the examples disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of various examples. The examples discussed herein were chosen and described in order to explain the principles and the nature of various examples of the present disclosure and its practical application to enable one skilled in the art to utilize the present disclosure in various examples and with various modifications as are suited to the particular use contemplated. The features of the examples described herein may be combined in all possible combinations of methods, apparatus, systems, and computer program products.