Adjusting host quality of service metrics based on storage system performance

A storage system has a QOS recommendation engine that monitors storage system operational parameters and generates recommended changes to host QOS metrics (throughput, bandwidth, and response time requirements) based on differences between the host QOS metrics and storage system operational parameters. The recommended host QOS metrics may be automatically implemented to adjust the host QOS metrics. By reducing host QOS metrics during times where the storage system is experiencing high volumes of workload, it is possible to throttle workload at the host computer rather than requiring the storage system to expend processing resources associated with queueing the workload prior to processing. This can enable the overall throughput of the storage system to increase. When the workload on the storage system is reduced, updated recommended host QOS metrics are provided to enable the host QOS metrics to increase. Historical analysis is also used to generate recommended host QOS metrics.

FIELD

This disclosure relates to computing systems and related device and methods, and, more particularly, to a method and apparatus for adjusting host QOS metrics based on storage system performance.

SUMMARY

The following Summary and the Abstract set forth at the end of this application are provided herein to introduce some concepts discussed in the Detailed Description below. The Summary and Abstract sections are not comprehensive and are not intended to delineate the scope of protectable subject matter, which is set forth by the claims presented below.

In some embodiments, a storage system has a QOS recommendation engine that monitors storage system operational parameters and generates recommended changes to host QOS metrics (throughput, bandwidth, and response time requirements) based on differences between the host QOS metrics and storage system operational parameters. The recommended host QOS metrics may be automatically implemented to adjust the host QOS metrics. By reducing host QOS metrics during times where the storage system is experiencing high volumes of workload, it is possible to throttle workload at the host computer rather than requiring the storage system to expend processing resources associated with queueing the workload prior to processing. This can enable the overall throughput of the storage system to increase. When the workload on the storage system is reduced, updated recommended host QOS metrics are provided to enable the host QOS metrics to increase. Historical analysis is also used to generate recommended host QOS metrics.

DETAILED DESCRIPTION

Aspects of the inventive concepts will be described as being implemented in connection with a storage system100connected to a host computer102. Such implementations should not be viewed as limiting. Those of ordinary skill in the art will recognize that there are a wide variety of implementations of the inventive concepts in view of the teachings of the present disclosure.

Some aspects, features and implementations described herein may include machines such as computers, electronic components, optical components, and processes such as computer-implemented procedures and steps. It will be apparent to those of ordinary skill in the art that the computer-implemented procedures and steps may be stored as computer-executable instructions on a non-transitory tangible computer- readable medium. Furthermore, it will be understood by those of ordinary skill in the art that the computer-executable instructions may be executed on a variety of tangible processor devices, i.e., physical hardware. For ease of exposition, not every step, device or component that may be part of a computer or data storage system is described herein. Those of ordinary skill in the art will recognize such steps, devices and components in view of the teachings of the present disclosure and the knowledge generally available to those of ordinary skill in the art. The corresponding machines and processes are therefore enabled and within the scope of the disclosure.

The terminology used in this disclosure is intended to be interpreted broadly within the limits of subject matter eligibility. The terms “logical” and “virtual” are used to refer to features that are abstractions of other features, e.g. and without limitation, abstractions of tangible features. The term “physical” is used to refer to tangible features, including but not limited to electronic hardware. For example, multiple virtual computing devices could operate simultaneously on one physical computing device. The term “logic” is used to refer to special purpose physical circuit elements, firmware, software, and/or computer instructions that are stored on a non-transitory tangible computer-readable medium and implemented by multi-purpose tangible processors, and any combinations thereof.

FIG. 1illustrates a storage system100and an associated host computer102, of which there may be many. The storage system100provides data storage services for a host application104, of which there may be more than one instance and type running on the host computer102. In the illustrated example the host computer102is a server with volatile memory106, persistent storage108, one or more tangible processors110, and a hypervisor or OS (Operating System)112. The processors110may include one or more multi-core processors that include multiple CPUs (Central Processing Units), GPUs (Graphical Processing Units), and combinations thereof. The volatile memory106may include RAM (Random Access Memory) of any type. The persistent storage108may include tangible persistent storage components of one or more technology types, for example and without limitation SSDs (Solid State Drives) and HDDs (Hard Disk Drives) of any type, including but not limited to SCM (Storage Class Memory), EFDs (Enterprise Flash Drives), SATA (Serial Advanced Technology Attachment) drives, and FC (Fibre Channel) drives. The host computer102might support multiple virtual hosts running on virtual machines or containers, and although an external host computer102is illustrated, in some embodiments host computer102may be implemented as a virtual machine within storage system100.

The storage system100includes a plurality of compute nodes1161-1164, possibly including but not limited to storage servers and specially designed compute engines or storage directors for providing data storage services. In some embodiments, pairs of the compute nodes, e.g. (1161-1162) and (1163-1164), are organized as storage engines1181and1182, respectively, for purposes of facilitating failover between compute nodes116. In some embodiments, the paired compute nodes116of each storage engine118are directly interconnected by communication links120. As used herein, the term “storage engine” will refer to a storage engine, such as storage engines1181and1182, which has a pair of (two independent) compute nodes, e.g. (1161-1162) or (1163-1164). A given storage engine118is implemented using a single physical enclosure and provides a logical separation between itself and other storage engines118of the storage system100. A given storage system100may include one or multiple storage engines118.

Each compute node,1161,1162,1163,1164, includes processors122and a local volatile memory124. The processors122may include a plurality of multi-core processors of one or more types, e.g. including multiple CPUs, GPUs, and combinations thereof. The local volatile memory124may include, for example and without limitation, any type of RAM, and in some embodiments is used to implement a cache for processors122. Each compute node116may also include one or more front-end adapters126for communicating with the host computer102. Each compute node1161-1164may also include one or more back-end adapters128for communicating with respective associated back-end drive arrays1301-1304, thereby enabling access to managed drives132.

In some embodiments, managed drives132are storage resources dedicated to providing data storage to storage system100or are shared between a set of storage systems100. Managed drives132may be implemented using numerous types of memory technologies for example and without limitation any of the SSDs and HDDs mentioned above. In some embodiments the managed drives132are implemented using NVM (Non-Volatile Memory) media technologies, such as NAND-based flash, or higher-performing SCM (Storage Class Memory) media technologies such as 3D XPoint and ReRAM (Resistive RAM). Managed drives132may be directly connected to the compute nodes1161-1164using a PCIe (Peripheral Component Interconnect express) bus, or may be connected to the compute nodes1161-1164, for example, by an IB (InfiniBand) bus or IB fabric switch136.

In some embodiments, each compute node116also includes one or more CAs (Channel Adapters)134for communicating with other compute nodes116directly or via an interconnecting fabric136. An example interconnecting fabric may be implemented using InfiniBand.

Each compute node116may allocate a portion or partition of its respective local volatile memory124to a virtual shared “global” memory138that can be accessed by other compute nodes116, e.g. via DMA (Direct Memory Access) or RDMA (Remote Direct Memory Access) such that each compute node116may implement atomic operations on the local volatile memory124of itself and on the local volatile memory124of each other compute node116in the storage system100.

The storage system100maintains data for the host applications104running on the host computer102. For example, host application104may write host application data to the storage system100and read host application data from the storage system100in order to perform various functions. Examples of host applications104may include, but are not limited to, file servers, email servers, block servers, and databases.

Logical storage devices are created and presented to the host application104for storage of the host application data. For example, as shown inFIG. 1, in some embodiments a production device140and a corresponding host device142are created implemented using InfiniBand.

Each compute node116may allocate a portion or partition of its respective local volatile memory124to a virtual shared “global” memory138that can be accessed by other compute nodes116, e.g. via DMA (Direct Memory Access) or RDMA (Remote Direct Memory Access) such that each compute node116may implement atomic operations on the local volatile memory124of itself and on the local volatile memory124of each other compute node116in the storage system100.

The storage system100maintains data for the host applications104running on the host computer102. For example, host application104may write host application data to the storage system100and read host application data from the storage system100in order to perform various functions. Examples of host applications104may include, but are not limited to, file servers, email servers, block servers, and databases.

Logical storage devices are created and presented to the host application104for storage of the host application data. For example, as shown inFIG. 1, in some embodiments a production device140and a corresponding host device142are created to enable the storage system100to provide storage services to the host application104. The host device142is a local (to host computer102) representation of the production device140. Multiple host devices142associated with different host computers102may be local representations of the same production device140. The host device142and the production device140are abstraction layers between the managed drives132and the host application104. From the perspective of the host application104, the host device142is a single data storage device having a set of contiguous fixed-size LBAs (Logical Block Addresses) on which data used by the host application104resides and can be stored. However, the data used by the host application104and the storage resources available for use by the host application104may actually be maintained by one or more of the compute nodes1161-1164at non-contiguous addresses in shared global memory138and on various different managed drives132on storage system100.

In some embodiments, the storage system100maintains metadata that indicates, among various things, mappings between the production device140and the locations of extents of host application data in the shared global memory138and the managed drives132. In response to an IO (Input/Output) command146from the host application104to the host device142, the hypervisor/OS112determines whether the IO146can be serviced by accessing the host computer memory106. If that is not possible then the IO146is sent to one of the compute nodes1161-1164to be serviced by the storage system100.

In the case where IO146is a read command, the storage system100uses metadata to locate the commanded data, e.g. in the shared global memory138or on managed drives132. If the commanded data is not in the shared global memory138, then the data is temporarily copied into the shared global memory138from the managed drives132and sent to the host application104via one of the compute nodes1161-1164. In the case where the IO146is a write command, in some embodiments the storage system100copies a block being written into the shared global memory138, marks the data as dirty, and creates new metadata that maps the address of the data on the production device140to a location to which the block is written on the managed drives132. The shared global memory138may enable the production device140to be reachable via all of the compute nodes1161-1164and paths, although the storage system100can be configured to limit use of certain paths to certain production devices140.

In some embodiments, the storage system100presents storage volumes as TLUs (Thin Logical Units). A TLU is a logical construct which enables the physical drives132of the storage system100to be abstracted from the host applications104. A logical unit is “thin”, as that term is used herein, when actual physical capacity of drives132is only allocated to the TLU as needed. For example, a TLU may be presented to the host application104as having an available size of 1T(terabyte). However, if the filesystem stored in the TLU is only 0.5 T in size, the storage system100will only allocate 0.5 T of physical storage resources on drives132to the TLU. Thus, the amount of physical storage resources allocated to a TLU will increase and decrease over time as the amount of data stored on the TLU changes over time. Within the physical storage resources, Data Devices (TDATs) are used to store the data, in which a given TDAT may be formed of a redundant group of physical disk drives, i.e. a TDAT may be formed from a RAID group of disk drives132that store blocks of data within the storage array130.

Different storage resources may have different IO characteristics. Storage resource132with similar IO characteristics are grouped together to form storage pools. Storage groups170(seeFIG. 2) are created within the storage pools. TLUs are allocated physical storage resources from a selected storage pool based on intended service level objectives for the data contained in the filesystem maintained by the TLU. The service level objectives are set by host computer102as host QOS metrics155. Different production devices140may have different service level objectives and, accordingly, different production devices140may be located in different storage groups170.

As shown inFIG. 1, in some embodiments one of the applications executing on the host computer102is a storage system management application150. The storage system management application150enables a customer to set host QOS (Quality Of Service) metrics155on the storage system100. Depending on the implementation, the host QOS metrics155may include multiple parameters of storage system100operation. Example host QOS metrics155may include, for example throughput, bandwidth, and response time. Throughput specifies the number of IOPS (IO operations per second) that the storage system100should provide. Bandwidth (MB/second) specifies the amount of front-end or back-end resources that should be allocated, such as on the front-end adapter126, back-end adapter128and/or fabric136. Response time specifies (ms) specifies the maximum amount of time the storage system100should take to respond to an IO operation. Numerous host QOS metrics155may thus be specified, and different host QOS metrics155may be set for different storage groups170or host devices142.

FIG. 2is a functional block diagram showing aspects of the QOS management aspects of the host computer and storage system ofFIG. 1in greater detail, according to some embodiments. As shown inFIG. 2, in some embodiments the storage system100has a QOS recommendation engine160configured to monitor operation of the components of the storage system and generate recommended host QOS metrics162. For example, the QOS recommendation engine160may monitor operation of the front end adapter126, CPU122, fabric interface134, and other operational aspects of the storage system100, to learn how much of each storage system100resource is being used by each storage group170.

Rather than setting host QOS metrics155as discrete static values per storage group170, the recommended host QOS metrics162are provided to the storage system management application150on host computer102, to enable the host computer102to adjust the host QOS metrics155. This enables host QOS metrics to be set based on a learned time series composite function within the storage system100, with lower and upper bound values set per storage group170.

In some embodiments, storage system100run time is divided into windows, and for each time window a QOS (Quality Of Service) recommendation engine160analyzes the internal performance metrics for each storage group170. The actual performance data of how the storage system100is performing is compared with the set of host QOS metrics155that has been set by the host computer102on those same storage groups170, to determine if there are any mismatches between the host QOS metrics155that have been set by the customer, and the actual operation of the storage system100. Where there is a difference between the host QOS metrics155and the performance that the storage system100is actually able to provide, the QOS recommendation engine160generates recommended host QOS metrics162that describe changes that should be made to change host QOS metrics155on the storage groups170for respective time windows. Aligning host QOS metrics155with storage system100performance increases overall performance of the storage system100. Specifically, by reducing the host QOS metrics155during periods where the storage system100is experiencing a high workload, the host computer102will reduce the number of IOs provided to the storage system100. This allows the storage system100to use more of its resources to process lOs rather than using its resources to store lOs prior to processing, which improves overall storage system100performance by aligning storage system100resources with high priority storage groups170.

Setting correct host QOS metrics155on storage groups170is necessary to enable a storage system100to serve different mixed workloads across multiple tenants (multiple host computers). Some types of workloads on the storage system100are moderately predictable, such as storage system100workloads associated with data replication and backup operations. Other types of workloads, such as storage system100workloads associated with on-line transaction processing, depends on business operations which makes these types of workloads difficult to predict. Typically, customers set discrete host QOS metrics155for on-line transaction processing storage groups170, replication storage groups170, and backup storage groups170. Since on-line transaction processing workload is often unpredictable, setting accurate host QOS metrics155for the on-line transaction processing storage group170can be difficult. If the host QOS metrics155for this storage group170are set to aggressively, this will result in over-allocation of storage system100resources to this storage group170.

In some storage systems100, enforcing host QOS metrics155is quite costly. Fiber channel or NVME Driver Interface threads in the front-end adapter126often work at the port level and don't adhere to host QOS metrics155. This means that the driver thread will continue to pull new command requests (IOs) from Fibre Channel (front end) ports and allocate local memory to the command requests. Additionally, in some embodiments, each command request involves a jorecord, Input/Output Control Block (IOCB) driver resources, Fibre Channel (FC) exchange control block resources, and miscellaneous store and forward buffers etc.

Additionally, each new command is added to the driver internal scheduling queue, and eventually to the host adapter126QOS priority queue management. This means that the host adapter126and other layers will spend many CPU cycles to repeatedly poll to check if IO's are getting throttled according to the host QOS metrics155. These memory and CPU resources, therefore, cannot be used for other important high priority storage groups170as well. This means that a storage groups170with lower host QOS metrics155can end up taking more memory and CPU cycles in preference to other higher priority storage groups170, which is counter intuitive. Accordingly, it would be advantageous to provide a way to address this internal storage system100resource wastage problem that occurs when the host QOS metrics155are set too high on one or more storage groups170, given the ability of the storage system100to process its current workload. In some embodiments, it would be advantageous to provide a method to throttle lOs from the host computer102with exceptionally reduced cost.

Host QOS metrics155are often set statically by ad hoc and heuristic processes, or based on past-experience by storage administrators. Static and discrete host QOS metrics155are too restrictive and are implemented using a manual process, which generally doesn't change with time based on the dynamic workload on the storage system100. Further, the storage system100does not provide feedback to the host computer102, causing storage system100resources to be underutilized or overutilized across different workloads and across different times on different storage groups170. The host QOS metric155mismatches across different storage groups170can cause resources of the storage system100to be incorrectly allocated, which reduces overall throughput of the storage system100. For example, in time window T1, storage group1701might need more bandwidth compared to storage group1702. However, with statically defined host QOS metrics155, the storage system100may be unable to allocated additional resources to storage group1701thus reducing the overall performance of the storage system100. Accordingly, it would be advantageous to enable the storage system100to participate in adjusting host QOS metrics155and make recommendations as to how the host QOS metrics155should be changed. In some embodiments, based on an internal workload analysis on the storage groups170, in a time window T, the storage system100determines which storage groups170should be allocated additional storage system100resources, and makes recommendations as to how the host QOS metrics155should be adjusted.

In some embodiments, for each storage group170, there are two types of QOS metrics: host QOS metrics155which are set by the host computer102, and recommended host QOS metrics162that are determined by the storage system100.

As shown inFIG. 1, in some embodiments the host QOS metrics155are set using the storage system management application150. The host QOS metrics155specify the high-level parameters that must be met by the storage system100for IO operations on particular storage groups170.

The recommended host QOS metrics162are learned by the storage system100based on workload analysis of storage system100across different time windows. In some embodiments, the recommended host QOS metrics162are determined by a QOS recommendation engine160, and are based on a QOS recommendation function. The recommendation function, referred to herein using the letter “R”, in some embodiments is a composite function including a combination of N functions, such as F1, F2, F3, F4. . . FN. Equation 1, set forth below, shows an example recommendation function R:

In equation 1, R is a composite function having three terms, R=F1+F2+F3. Function F1is the difference between the host QOS metric155that the customer specified for the storage system100response time for the given storage group170, and the response time the storage system100is actually seeing, internally, for the same storage group170during a given time window T. Function F2is the difference between the host QOS metric155that the customer specified for the number of input/output operations per second (IOPs) for the given storage group170, and the number of IOPs the storage system100is actually seeing, internally, for the same storage group170during the given time window T. Function F3is the difference between the host QOS metric155that the customer specified for the bandwidth for the given storage group170, and the bandwidth the storage system100is actually seeing, internally, for the same storage group170during the given time window T. Thus, the recommendation function R, in some embodiments, enables any mismatch between host QOS metrics155and actual storage system100performance to be quantified by the QOS recommendation engine160.

Each component of the recommendation Function R (F1, F2and F3), are weighted separately as needed by the customer to provide different priorities for response time, the number of IOs per second, or bandwidth. For example, by setting the weight factor for response time to 1 (Weight 1=1), and setting the weighting factors for IOPs and bandwidth to 0.1 (Weight2=0.1; Weight 3=0.1), the recommendation function R would provide greater emphasis on the storage system100response time when making recommendations for adjustment to host QOS metrics155. In some embodiments, by default, each weight is set to 1 i.e. all weights are same.

The output of the recommendation function R is a set of recommended host QOS metrics162which, in some embodiments, are used to automatically adjust the host QOS metrics155for the upcoming time window. For example, as shown inFIG. 2, in some embodiments the recommended host QOS metrics162are passed to storage system management application150and used by storage system management application150to change host QOS metrics155for an upcoming time interval. In other embodiments, the output of the recommendation function R is used to make a recommendation to the storage system management application150to prompt the customer to make changes to the host QOS metrics155. Additionally, the values of the components of the recommendation function R indicate how the host QOS metrics155should be changed. Specifically, the value of the Fl component indicates the amount that the host QOS metric155associated with response time should be changed, the value of the F2component indicates the amount that the host QOS metric155associated with IOPs should be changed, and the F3component indicates the amount that the host QOS metric155associated with bandwidth should be changed. In some embodiments, the values of the components of the recommendation function (F1, F2, F3) are provided to storage system management application150.

In some embodiments, additional storage system100workload features are similarly used to create additional weighted components R=(F4, F5, . . . FN) that are also included in the recommendation function R(F1, F2, F3, F4, F5, . . . FN) that is used by QOS recommendation engine160to generate the recommended host QOS metrics162.

One example of an additional component (F4) that may be included in the recommendation function R is whether prefetch has been turned on/off on cache124for a given storage group170in this given time window. If customer chooses prefetch to be switched off for given storage group170, this decision will reduce storage system100fabric bandwidth consumption that can then be used by other storage groups170, but may also lead to read misses for sequential host IOs146. Each storage system100feature, such as whether prefetch is turned on for the cache, the amount of storage system100resources that are consumed in connection with features such as synch/async replication of data to other storage systems100, log scrubbers, and low priority tasks, is computed with respect to CPU cycle consumption, memory consumption and fabric bandwidth consumption per storage group170, and can be controlled by the customer as each storage system100feature has its own weights per storage group170.

Thus, in some embodiments, the recommendation function R (IOPs, response time, and Bandwidth Recommendation for a given Storage Group(SG1_Time_window_1) during a given Time Window)=

F1 (storage system response time, customer set response time)*Weight1+

F4(storage group (SG) level Prefetch CPU consumption time series)*Weight4+

F5(SG level Prefetch fabric bandwidth consumption time series)*Weight5+

F6(SG level Asynchronous Memory consumption time series)*Weight6+

F7(SG level Asynchronous CPU cycle consumption time series)*Weight7+

F8(SG level data relocation feature CPU cycle consumption time series)*Weight8 . . .

The customer can choose to switch off any storage system100feature per storage group170by setting the weights of the respective feature to zero. In some embodiments, by default each feature has equal priority, and weights 1-8 are all set to 1. Other default values may be used, depending on the implementation.

In some embodiments, time is divided into N windows per hour. By default, each time window is IO minutes, but the duration of the time windows can be user defined. During each time window, a time series is generated for the number of IOs per second, storage system100response time, and storage system100internal bandwidth consumption, for each storage group170(or device140). The time series is used to find lower and upper bound values by using exponential moving averages. These values are then compared with customer set host QOS metrics155to find if the storage system100is over utilized or underutilized. For each storage group170, recommended host QOS metrics162such as prefetch, asynchronous replication, data relocation, scan, low priority tasks and scrubbers are computed. Recommended host QOS metrics162, in some embodiments, are defined by percentage of CPU cycles, percentage memory usage, and percentage of bandwidth consumption, and are calculated per storage group170.

The recommended host QOS162metrics are used to populate a data structure300(seeFIG. 3) containing information about the amount (percentage) of the storage system100resources each feature is consuming. In some embodiments, the data structure300is provided to storage system management application150to enable the storage administrator to be able to access this information. The storage system100resource cost per feature is aggregated across all host computers102for each storage group170. Similarly, a time series is also built for each storage system100internal resource consumption with its own lower and upper bound in a given time window.

FIG. 3is a functional block diagram of an example data structure300containing example storage system100performance data for use by a QOS recommendation engine160to implement or recommend QOS adjustments to host QOS metrics155based on storage system100performance, according to some embodiments. As shown inFIG. 3, during a time window, the QOS recommendation engine160collects data on numerous features. For example, one of the system performance parameters that may be monitored by the QOS recommendation engine160is the “Aggregate Host RT” (Response Time). In the example data structure300shown inFIG. 3, an example storage system100internal lower bound response time of 0.1 ms was recorded during the time interval, and an example storage system100upper bound response time of 0.5 ms was recorded during the time interval. The customer set host QOS metric155, by contrast, was set to 0.1 ms. This indicates that the host QOS metric155is set to a value (0.1 ms) which is not consistently being met by the storage system100.

Additionally, as shown inFIG. 3, the QOS recommendation engine160also collects “Aggregated Host IOPS” (Input/Output Per Second) during the time interval. In the example data structure300shown inFIG. 3, the storage system100internal lower bound value was 1 Gbps (gigabyte per second) and the storage system100internal upper bound value was 8 Gbps. The host QOS metric155for aggregated IOPs was set to 6 Gbps. This indicates that the storage system100is overused, or that the host QOS metrics155for this parameter is not set high enough.

The combination of measured vs requested response time and measured vs requested bandwidth indicates that the storage system100is either over-utilized or that the host QOS metrics155for the storage group170are not set properly. To enable the storage system100to more consistently meet the host QOS metrics155, a recommendation is provided by the QOS recommendation engine160that the host QOS metric155for bandwidth be increased from 6 Gbps to 8 Gbps. This will enable the host computer102workload on the storage group170to be given higher priority from a bandwidth perspective, to enable the storage system100to more closely achieve the 0.1 ms response time specified in the host QOS metrics155.

Looking at the line entry for “Array Fabric Bandwidth Usage”, it can be seen that the maximum storage system100bandwidth used during the time interval was 16 Gbps. However, in this example the host computer102is connecting to a front-end adapter126on the storage system100that has an 8 Gbps speed, which is reducing the storage system100response time. The recommendation by the QOS recommendation engine160is that the host computer102should use a 16 Gbps front-end adapter126or use multiple front-end adapters126when communicating with the storage system100. Specifically, while changing the host QOS bandwidth metric155from 6 Gbps to 8 Gbps will help slightly, the fact that the maximum fabric bandwidth was 16 Gbps for this storage group indicates that the host computer102should either use a front-end adapter126with a larger bandwidth to address this performance issue or to distribute the host workload on storage group1701across more front-end adapters126. Also, it can be seen from the “Prefetch” entry that prefetch has been turned on, and from the “read” entry, it appears that 30% of the fabric bandwidth is being consumed by read operations, which indicates that large sequential reads are in progress.

In some embodiments the QOS recommendation engine160recommends changes to host QOS metrics155per storage group170, per time interval, to cause the host computer102to adjust its QOS metrics155to better match the capabilities of the storage system100.

In some embodiments, the QOS recommendation engine160does a live technical analysis on time series charts for host IOPS, host response time, host FC bandwidth, storage system100internal fabric bandwidth, storage system100memory consumption, prefetch CPU cycle consumption, prefetch fabric bandwidth consumption, asynchronous replication CPU cycle consumption, asynchronous replication fabric bandwidth consumption, and storage system100CPU cycle consumption, per storage group170, to find the current storage burst (trend) activities on a given storage group170. In some embodiments, this analysis is implemented using exponential moving averages for every time window. The time windows may be, for example, ten minutes in length or some other amount of time depending on the implementation. Lower and upper bound values from these time series values are compared with customer set host QOS metrics155to find average percentage of over utilization or underutilization of storage system100resources. The QOS recommendation engine160then provides a set of recommended changes to the host QOS metrics155. The recommended changes to host QOS metrics155may be output as normalized values or as in percentages. In some embodiments, the time series values per storage group170are saved for subsequent historical analysis.

In some embodiments, QOS recommendation engine160also does historical technical analysis to find any trend or seasonality in time series charts for host IOPS, host RT, host FC Bandwidth, internal fabric bandwidth, memory consumption, and CPU cycle consumption per storage group170across different time (windows) and days using double and triple exponential moving averages or uses a learning process161(seeFIG. 2. In some embodiments, the learning process161is a learning process such as a such as a long short-term memory model (LSTM neural network).

Simple moving average analysis is also done on live and historical time series for host IOPS, RT and bandwidth, which can be used to determine lower bound values for the recommended host QOS metrics162.

In some embodiments, the upper bound for recommended host QOS metrics162per storage group170comes from Max (weight_1*LSTM prediction, weight_2*exponential moving averages prediction, weight_3*host specified QOS metrics155). Max upper bound values are weighted based on customer expectations. The customer can elect to use this dynamic recommendation feature or use only customer defined host QOS metrics155or a combination of both host QOS metrics155and the recommended host QOS metrics162across different times. In some embodiments, values for weight_1, weight_2 and weight_3 determines the customer preferences.

The recommendation engine160provides, as output, recommended host QOS metrics162in the form of recommendations on both lower(*Weight_A) bound host QOS metrics155and higher(*Weight_B) bound host QOS metric155for each storage groups170to host computer102. Lower and higher bounds are also weighted here, so the customer can define preferences as needed. If a customer goes with weight_B always, then it might under-utilize the storage system100resources i.e. some storage groups170will get more storage system100resources, even when they are not needed thus causing performance issues for other storage groups170. If a customer gives equal preference for both lower and upper bound weights, then the storage system100does better scheduling by moving storage system100resources between storage groups170, e.g. from to storage group1701to storage group1702(or other storage groups170) as needed, while still maintaining storage group1701host QOS metrics155to meet its lower bound values. If the workload on storage group1701increases, then the storage system100dynamically moves storage system100resources from other low priority storage groups170and moves those resources to storage group1701. Storage groups170are prioritized based on response time, IOPs and bandwidth requirement as set by the host QOS metrics155.

The QOS recommendation engine160provides recommended host QOS metrics162per storage group170per time window. Having these recommended host QOS metrics162enable the host computer102to throttle some host applications104(like oracle, SAP, DB, OLTP, OLTA) when necessary, to cause the storage system100to devote more of the storage system100resources to higher priority applications (or storage groups170), thus both the host computer102and the storage system100cooperatively cause storage system100resources to be allocated to the correct storage groups170, to service workloads for the correct host applications104, at the right time.

In some embodiments, each time series (trajectory) is split into multiple time windows (from few minutes to few hours per window). By analyzing how much storage system100resources (example CPU cycles, memory, bandwidth) each storage system100feature (example multi tenancy, compression, data relocation) takes in a given time window per storage group170, it is possible to understand what percentage of these resources are relevant and irrelevant with respect to host QOS metrics155. When additional storage system100resources are available for use by the host computer102, the QOS recommendation engine160quantifies the amount of available system resources and provides the host computer102with updated recommended host QOS metrics162, to enable the host computer102to increase the workload on the storage system100. If multi tenancy feature takes more CPU cycle in a given window for a given storage group170, it means that the storage system100is spending a large amount of CPU cycles just to throttle the IO's in the front end (host adapter126) layer unnecessarily, instead of using these resources to service10s. Excessive unnecessary throttling means host QOS metrics155were probably not set right in the first place by customers on those storage groups170, or that the storage administrators underestimated the host QOS metrics155for given storage group170.

FIG. 4is a graph showing fluctuation of host QOS metrics over time based on the output of a QOS recommendation engine, according to some embodiments. Rather than simply utilizing fixed host QOS metrics155and holding the host QOS metrics155at a constant value, in some embodiments the QOS recommendation engine160automatically adjusts the host QOS metrics155according to demand on the storage system100, so that the host QOS metrics155are not set too high relative to a level of service that the storage system100is able to provide. Accordingly, the host QOS metrics155are aligned with a level of service that the storage system100is able to provide, given other demands on storage system100resources. By adjusting the host QOS metrics155, it is possible to cause the number of IO146per second from the host computer102to the storage system100to more closely align with the amount of work that the storage system100is able to perform. By reducing the amount of excess work provided by the host computer102to the storage system100, it is possible to reduce the amount of CPU cycles used by the storage system100to manage that excess workload from the host computer102, to thereby free those resources for use in connection with processing IO operations.

In the example shown inFIG. 4, in a first time-interval the QOS recommendation engine160recommends that the host QOS metric155for IOPS be set to 20K. At a second time interval, the QOS recommendation engine160recommends that the host QOS metric155for IPS be set to 30K. At a third time interval, the QOS recommendation engine160recommends that the host QOS metric155for IOPS be set to 10K. By dynamically changing the host QOS metric155in this manner, the host QOS metrics155are able to be set to match the capabilities of the storage system100to reduce overhead associated with setting the host QOS metric155in excess of the capabilities of the storage system100for the storage group170. The time interval shown in the graph ofFIG. 4can be dynamically determined based on current conditions at the storage system100or, may be based on historical traffic patterns.

FIG. 5is a flow chart of an example method of adjusting host QOS metrics based on storage system performance, according to some embodiments. In some embodiments, the method includes collecting storage system100performance data for storage groups170during a time window (block500). The collected storage system100performance data is then compared with host QOS metrics155for the storage group170(block505). In some embodiments, comparing the storage system100performance data with host QOS metrics155is implemented by a QOS recommendation engine160using the recommendation equation R described in greater detail above.

A comparison is then made to determine whether the recommended host QOS metrics162output by the QOS recommendation engine160are different than the host QOS metrics155. Optionally, as shown inFIG. 5, a threshold may be used to determine if the recommended host QOS metrics162are sufficiently different than the host QOS metrics155. The threshold may be set to zero if it is desired to always output the recommended host QOS metrics162.

If the recommended host QOS metrics162are sufficiently different from the host QOS metrics155(a determination of YES at block510), the host QOS metrics155on the storage group170are automatically or manually adjusted (block515). If the recommended host QOS metrics162are not sufficiently different from the host QOS metrics155(a determination of NO at block510) the process returns to continue collecting storage system100performance data for the storage group170during a subsequent time period. This process iterates for each time period, and for each storage group170.

As shown inFIG. 5, in some embodiments the storage system performance data that is collected for each storage group170in block500is also used by the QOS recommendation engine160to perform historical analysis (e.g. using learning process161) to determine trends in storage system100performance (block520). If a trend is detected (a determination of YES at block525), the historical analysis can be used to automatically or manually adjust the host QOS metrics155(block515) to account for the anticipated changes in storage system100performance. If no trend is detected, the process returns to continue collecting storage system100performance data for the storage group170during a subsequent time period. This process iterates for each time period, and for each storage group170.

The methods described herein may be implemented as software configured to be executed in control logic such as contained in a Central Processing Unit (CPU) or Graphics Processing Unit (GPU) of an electronic device such as a computer. In particular, the functions described herein may be implemented as sets of program instructions stored on a non-transitory tangible computer readable storage medium. The program instructions may be implemented utilizing programming techniques known to those of ordinary skill in the art. Program instructions may be stored in a computer readable memory within the computer or loaded onto the computer and executed on computer's microprocessor. However, it will be apparent to a skilled artisan that all logic described herein can be embodied using discrete components, integrated circuitry, programmable logic used in conjunction with a programmable logic device such as a Field Programmable Gate Array (FPGA) or microprocessor, or any other device including any combination thereof. Programmable logic can be fixed temporarily or permanently in a tangible non-transitory computer readable medium such as random-access memory, a computer memory, a disk, or other storage medium. All such embodiments are intended to fall within the scope of the present invention.