Dynamic performance virtualization for disk access

A storage control system includes performance monitor logic configured to track performance parameters for different volumes in a storage array. Service level enforcement logic is configured to assign target performance parameters to the different volumes and generate metrics for each of the different volumes identifying how much the performance parameters change for the different volumes responsive to changes in the amounts of tiering media allocated to the different volumes. Resource allocation logic is configured to allocate the tiering media to the different volumes according to the performance parameters, target performance parameters, and metrics for the different volumes.

BACKGROUND

Block-level storage in a disk storage array is organized as volumes of logical units (LU). Servers access these disk storage array volumes as blocks. The major metrics for these volumes are:

CAPACITY—amount of available storage (in bytes);

IOPs—Input/Output operations per second (that the volume can handle);

THROUGHPUT—data rate for a particular volume.

For reference, a typical disk storage array volume using serial Small Computer System Interface (SCSI) disks may have parameters as follows:

Access to a disk storage array is relatively slow compared to Dynamic Random Access Memory (DRAM) or Solid State Flash (Flash) memory. As mentioned above, a memory access to disk can take several milliseconds while RAM accesses are on the order of nano-seconds and Flash memory accesses are on the order of microseconds.

DETAILED DESCRIPTION

Several preferred examples of the present application will now be described with reference to the accompanying drawings. Various other examples are also possible and practical. This application may be exemplified in many different forms and should not be construed as being limited to the examples set forth herein.

Referring toFIG. 1, a performance virtualization appliance (PVA)14is located between one or more clients10and a disk storage array24. The clients10may be servers, personal computers, terminals, portable digital devices, routers, switches, or any other wired or wireless computing device that needs to access data on disk storage array24. In one embodiment, the PVA14and the disk storage array24are stand-alone appliances, devices, or blades.

In one embodiment, the clients10, PVA14and disk storage array24might be coupled to each other via wired or wireless Internet connections12. In another embodiment, the clients10may access one or more of the disks in disk storage array24over an internal or external data bus. The disk storage array24in this embodiment could be located in the personal computer or server10, or could also be a stand-alone device coupled to the computer/server10via a fiber channel SCSI bus, Universal Serial Bus (USB), or packet switched network connection.

The PVA14contains one or more processors that operate as a virtualization controller16. A tiering media50contains different combinations of Flash memory20and DRAM Memory22that may have faster access speeds than the disk storage array24.FIG. 2shows an alternative view of the tiering media50located in the appliance14that includes the Flash media20and DRAM media22. The virtualization controller16receives read and write operations from clients10directed to particular storage volumes26in disk storage array24. In one embodiment, the storage volumes26are different groups of one or more disks, storage blocks, or any other grouping of memory that may be associated with a particular Service Level Agreement (SLA). An SLA is a storage access performance level that is expected when accessing an associated storage volume in the disk storage array24. Throughout the specification the terms storage volume and disk will be used interchangeably.

The virtualization controller16uses different combinations of the Flash memory20and the DRAM22, in conjunction with disk storage array24, to provide dynamic performance virtualization for the different storage volumes26. For example, the virtualization controller16can dynamically or statically change the Input/Outputs (I), Latency (L), and Throughput (T) observed by the clients10when accessing the different storage volumes26in disk storage array24. In an alternative embodiment, other measures of performance derived from I, L, and T may be used by the virtualization controller.

Different combinations of the performance parameters I, L, and T are tracked by the virtualization controller16and used for enforcing SLAs for different clients10and/or storage volumes26. The amount of Flash20, DRAM22, and disk24allocated to a particular storage volume26can be dynamically varied to adaptively track the SLAs assigned to those storage volumes.

Different storage volumes26A,26B, and26C can be assigned to different client data. For example, data volume26A (V1) might be associated with the reservation data for an airline, data volume V2may be associated with the flight schedule data for the same airline, and data volume V3may be associated with an on-line retail store for a different client. Of course, any type of client10and any type of data can be associated with the different storage volumes26.

Different SLA parameters are assigned to the different storage volumes26. For example, volume V1may be assigned a SLA value SLA1, storage volume V2may be assigned a different SLA value SLA2, and volume V3may be assigned another different SLA value SLA3. These are just examples, and any combination or gradation of SLA values can be assigned to the different volumes26.

The virtualization controller16allocates different amounts18of Flash memory20and DRAM22to the different storage volumes26according to the associated SLA values. In one example, volume26A may have a highest SLA value SLA1, volume26B may have a lower SLA value SLA2, and volume26C may have an even lower SLA value SLA3. A relatively high SLA value could correspond with any combination of a relatively large number of Input/Outputs (I), a relatively small Latency (L), and/or a relatively large Throughput (T). A lower SLA value may correspond to a smaller number of IOs (I), a relatively larger Latency (L), and/or a relatively smaller Throughput (T).

As mentioned above, Flash20, DRAM22, and disk storage array24have different access speeds, with DRAM22generally having the fastest access time, Flash having a next fastest access time, and disk storage array24having the slowest access time. It should also be understood that any other type of storage can be used in tiering media50. For example, other types of Random Access Memory (Such as Ferroelectric RAM or Phase-change memory) or other relatively fast disk or solid state memory devices can also be used in tiering media50.

The virtualization controller16allocates different amounts of DRAM22and Flash20to the storage volumes26to meet the SLA values. The memory allocations18are mappings stored in tables in the appliance14that indicate what addresses in Flash20and DRAM22are used in conjunction with the different storage volumes26. For example, a relatively large amount of DRAM memory (R) may be identified in memory allocation18A for storage volume26A, since DRAM has a faster access time than Flash20or disk24, in order to meet the SLA for volume26A.

The volume26B has a lower SLA value SLA2. Therefore, volume26B may have a memory allocation18B with relatively less DRAM22and/or Flash20, compared with volume26A. Volume26C has a lowest. SLA value SLA3and accordingly may be assigned a memory allocation18C with little or no DRAM memory22and a relatively small amount of Flash memory20. These examples are for illustrative purposes and any other combinations of DRAM22, Flash20, and disk24can be allocated to different volumes26.

Dynamic Allocation

The virtualization controller16uses the memory allocations18A,18B, and18C when accessing volumes26A,26B, and26C, respectively. Based either at a particular time, on a particular access or access pattern to one or more block in a particular volume26, the controller16may pre-fetch other blocks for the same storage volume26into DRAM22and Flash20.

For example, the SLA1for volume26A may be associated with an overall response time of 500 microseconds (μs). A particular percentage of DRAM22and a particular percentage of Flash20are allocated to volume26A that result in an overall client response time of around 500 μs. Whenever a particular block in volume26A is accessed, or at a particular time, or when a particular data access pattern is detected, one or more blocks for the volume26A may be loaded into the DRAM22and/or Flash20allocated to that storage volume. The DRAM22and/or Flash20are then used for any subsequent accesses to those particular blocks for that particular volume26A.

The controller16continuously monitors the number of reads and writes directed to volume26A that may either be directed to DRAM22, Flash20, and disk storage array24. The overall response time associated with a storage volume26might be slower than 500 μs when the allocation of DRAM22and Flash20to that particular storage volume is too low. Accordingly, the controller16may allocate more DRAM22and/or Flash20to that particular storage volume26A so that more data can be stored in the faster tiering media50, more reads and writes can be serviced by the faster DRAM22and Flash20, and the overall performance of storage volume26A operates more closely to the 500 μs SLA. In the event that all tiering resources are exhausted before reaching the volume SLA, the virtualization controller has achieved the best possible performance (best effort). Configuration may allow, in one embodiment, for this best effort using all available resources to be considered as satisfying the SLA.

Conversely, the proportion of reads and/or writes to DRAM22and Flash20for volume26A may be too large. This would correspond to an overall response time for volume26A that is faster than the 500 μs value for SLA1. In this situation, the virtualization controller16may dynamically de-allocate some of the DRAM22and Flash20previously allocated to volume26A. This would result in a fewer number of reads and writes to DRAM22and/or Flash20and slower overall memory access performance.

The particular blocks from the disk storage array24loaded into DRAM22and Flash20may also be dynamically or statically selected. The controller16may identify which blocks and/or storage volumes26are associated with a particular client10or SLA and are receiving the largest number of reads and writes. For example, a database in disk storage array24may have indexes that are frequently accessed when identifying customer records. The controller16may assign these storage volumes higher SLA values and accordingly load the blocks or volumes containing these heavily used indexes into DRAM22and/or Flash20to more efficiently utilize the memory resources for a particular client, SLA, or volume.

In another example, databases may use space on disk storage array24to temporarily hold data associated with a particular query. The data in this temp space is then repeatedly accessed by the client10. The controller16may detect a read and write signature associated with these temporary space volumes and store the identified volumes containing the temporary space in DRAM memory22and Flash memory20.

In time based dynamic allocation, the controller16may determine that particular volumes26are heavily accessed at different times of the day. For example, a particular client10may access particular storage volumes26from 9:00 am to 10:00 am. The controller16may automatically increase the SLA values for these storage volumes from 9:00 am to 10:00 am so that more data from these storage volumes can be pre-fetched and used in DRAM22and/or flash20from 9:00 am to 10:00 am. The controller16may then lower the SLA values for these volumes after 10:00 am when fewer memory accesses are expected and increased memory access performance is no longer required.

Thus, the overall efficiency of reducing memory access time is improved by allocating more relatively fast memory20and22to storage volumes with high SLA requirements and possibly reducing the SLA values for these storage volumes during periods of relatively low utilization.

Virtualization Controller

FIG. 3shows different operations or functional elements of the virtualization controller16. The operations may be performed by a programmable processor responsive to executed computer instructions or could be performed with logic circuitry, one or more programmable logic devices or an application specific integrated circuit (ASIC).

A performance monitor60tracks the different storage operations from the clients10to the storage volumes26in storage array24and determines different performance parameters for the different storage volumes26. Service level enforcement logic70determines when the different storage volumes are not meeting associated Service Level Agreements (SLA). Resource allocation logic80allocates the tiering resources20and22to the different storage volumes26inFIG. 1according to the SLA violations detected by the enforcement logic70.

FIG. 4shows information tracked in a storage monitor table65by the performance monitor60. The performance monitor60obtains the parameters in table65by monitoring the read operations received and sent back to clients10inFIG. 1. The storage monitor60uses information contained in the disk read operations to identify what disks26inFIG. 1are associated with the read operation. The information also indicate the number of blocks addressed in the read operations and the starting block of the operation.

A first timer (not shown) is used by the performance monitor60to track how long it takes particular disks26to respond to read operations. A second timer and a counter (not shown) are used to track how many read operations are completed by particular disks26over a predetermined time period. Another counter (not shown) is used to track the total number of blocks read during the predetermined time interval.

For the particular time intervals indicated in column65B of table65, the performance monitor60tracks the number of read operations performed on a particular disk or storage volume26(FIG. 1) in column65C. The total number of storage blocks read during those read operations is tracked in column65D, a total amount of time required to complete all of the read operations is tracked in column65E, and an average latency for each of the read operations identified in column65C is identified in column65F.

For example referring to the first row in table65, during a particular time interval1, there were 100 read operations to disk A and a total of 100,000 blocks read from disk A. The total amount of time required to read the 100,000 blocks was 100 milliseconds (msec) and on average, each read operation took 1 msec.

A second row of table65shows another record for the same disk or storage volume A for a different time interval2. For example, time interval1in the first row of table65may be 1 second. The time interval2in the second row of table65also may be 1 second, but may be taken at a different time. During time interval2, 300 read operations were made by the clients10to disk A and 300,000 blocks were read. The total amount of time required to read the 300,000 blocks was 900 msec. and the average time for each of the 300 read operations was 3 msec.

FIG. 5shows a table75that contains enforcement parameters used by the service level enforcement logic70. Column75A identifies the particular disk or storage volume26where enforcement is provided by logic70. Column75B identifies a target number of read operations that may be enforced for a particular disk. Column75C identifies a target number of read blocks that may be enforced for a particular disk or storage volume, and column75D identifies a target latency that may be enforced for a particular disk or storage volume. If a value in a particular column is zero, then that associated parameter may not be enforced for the associated disk.

For example, the first row in table75is used by the service level enforcement logic70to enforce an average latency of 0.5 msec on disk A. However logic70will not enforce a target number of read operations (column75B) or enforce a target number of read blocks (column75C) since these fields are zero. The second row in table75is used by the service level enforcement logic70to enforce a target number of 2000 read blocks and enforce a target latency of 0.5 msec for disk B. However, the logic70will not enforce a target number of read operations (column75B) on disk B. The Nth row in table75is used by the service level enforcement logic70to enforce a target number of 200 read operations and a target latency of 0.5 msec on disk N. However, logic70will not enforce a target number of read blocks on disk N.

The parameters in table75may be preconfigured based on empirical data related to the data read patterns of the different disks26in storage array24(FIG. 1). For example, a particular disk, it may be more important to pipeline a large number of read operations back-to-back. For this disk, the SLA may only specify a particular value in the target read operation column75B.

For another disk or storage volume, it may be more important to read a large number of blocks in a shortest amount of time. The SLA for this disk may specify a particular large value in the target read block column75C. Alternatively, overall latency of each read operation may be a primary criteria. The SLA for this disk may specify a relatively small value in column75D. Any combination of the above parameters may be specified for any combination of disks.

FIGS. 6-8describe in more detail how the service level enforcement logic70operates.FIG. 6shows one example of how service level enforcement is performed for disk A according to the 1 second time interval indicated in table65. In operation100, logic70calculates the number of read operations per second provided by disk A by dividing the number of read operations (100) in column65C by the 1 second time interval in column65B. In operation102, the number of blocks read per second is calculated by dividing the total number of blocks read for disk A in column65D (100,000) by the 1 second time interval. Operation104calculates the average read latency for disk A as 1 msec by dividing the total read latency in column65E (100 msec) by the number of read operations in column65C (100).

In operation106the enforcement logic70compares the tracked values with the target enforcement parameters in table75. For example, for disk A the number of read operations/sec=100 and the target read operations/sec in column75B for disk A is 0. No SLA has been violated since 100>0. The number of tracked read blocks/sec=100,000 for disk A and the target read blocks for disk A in column75C is 0. No service level agreement has been violated since 100,000>0.

The average latency for disk A over the 1 second time period was 1 msec. However the target average latency for disk A is 0.5 msec. Because 1 msec>0.5 msec, the enforcement logic70in operation108determines that the service level agreement for disk A has been violated. In response to the SLA violation indication, the resource allocation logic80inFIG. 3will allocate more tiering media50to disk A. The allocation process will be described in more detail below inFIGS. 9-13.

FIG. 7shows another example of how the service level enforcement is performed for disk B according to the measurements in the third row of table65inFIG. 4. In operation120, logic70calculates the number of read operations per second by dividing the number of read operations (50) by the 1 second time interval. In operation122, the number of blocks read per second is calculated by dividing the total number of blocks read for disk B in column65D (5,000) by the 1 second time interval. Operation124calculates the average read latency for disk B by dividing the total read latency in column65E of table65(25 msec) by the total number of read operations in column65C (50)=0.5 msec.

In operation126the enforcement logic70compares the derived values with the target enforcement parameters in table75. For disk B, the number of read operations/sec=50 and the target number of read operations/sec in column75B is 0. No service level agreement has been violated since 50>0. For disk B, the number of read blocks/sec=5,000 and the target read blocks for disk B in column75C is 2000. Since 5,000>2000, there is no violation of the SLA.

The average latency for disk B over the 1 msec time period is 0.5 msec and the target latency for disk B in column75D is also 0.5 msec. Because the measured average latency 0.5 msec≦the 0.5 msec target latency in table75, disk B currently does not violate the target latency in the SLA.

The enforcement logic70in operation128indicates that disk B does not currently violate the associated SLA. In response, the resource allocation logic80will not make any adjustments to the tiering media50currently allocated to disk B during the next allocation period.

FIG. 8shows another example of how SLA enforcement is performed for disk N according to the measurements in the Nth row of table65inFIG. 4. In operation140, the number of read operations (100) is divided by the 1 second time interval. In operation142, the total number of blocks read for disk N in column65D (50,000) is divided by the 1 second time interval. Operation144calculates the average read latency for disk N by dividing the total read latency in column65E of table65(50 msec) by the total number of read operations in column65C (100)=0.5 msec.

In operation146the enforcement logic70compares the derived values with the target parameters in table75. For disk N the measured number of read blocks/sec=50,000>the target number of read blocks for disk N=0. This is determined not to be a violation of the SLA. The average latency for disk N over the 1 msec time period is 0.5 msec which is ≦ than the target latency of 0.5 msec in column75D. This is also determined not to be a violation of the service level agreement by the enforcement logic70.

However, for disk N the total number of read operations/sec=100 and the target number of read operations/sec in column75B is 200. Because, 100 read ops/sec<200 read ops/sec, a service level agreement violation is indicated in operation148.

FIG. 9explains how the resource allocation logic80allocates tiering media to disks that have violated their SLAs in table75. Table85lists particular disks in column85A and a current amount of tiering media currently allocated by those particular disks in column85B. For example, the amount of Flash memory20and DRAM22currently allocated to disk A is indicated as 1000 Gigabytes (GB) in column85B. The values in column85B could be one value that represents a combined amount of Flash20and DRAM22allocated to the associated disk or there could include separate numbers identifying each different type of tiering media.

The maximum resource values in column85C indicate a limit on the amount of tiering resources that can be assigned to a particular disk or storage volume26. For example, based on the particular SLA associated with disk A, column85C indicates that a maximum of 2000 GB of tiering media50may be allocated to disk A. Again, there may be separate numbers in column85C for Flash20and DRAM22. Column85D indicates how much tiering media was added or removed during a last enforcement period. For example, during a last allocation session, disk A was allocated an additional 10 GBs of tiering media.

FIG. 10is a flow diagram that explains in more detail how the resource allocation logic80operates. A violation of a service level agreement for one or more disks is detected in operation160as described above inFIGS. 6-8. Upon detection of the violation, the current tiering resource allocation for that particular disk identified in column85B is compared with the maximum resource allocation identified in column85C. If the current amount of allocated resources is below the maximum allowable allocation in operation164, more tiering media may be allocated to the disk. Operation166then updates table85to reflect the additional allocation. Otherwise, no additional tiering media is allocated.

The tiering media50is allocated by assigning additional address ranges either in the Flash20and/or DRAM22to the associated disk or storage volume26(FIG. 1). The virtualization controller16or some other processing element in the appliance14(FIG. 1) uses the additional allocated tiering media50for prefetches, look-aheads, or any other type of local storage of data associated with that particular disk. For example, allocating more tiering media50to disk A, allows more data from disk A to be stored in the faster tiering media50. This can then result in an over reduction in read latency, an overall increase in the number of read operations that can be completed, or an increase in the number of blocks that can be read over a particular time period.

At least one example of particular use of the tiering media is described in co-pending application Ser. No. 12/605,160, filed Oct. 23, 2009, entitled BEHAVIORAL MONITORING OF STORAGE ACCESS PATTERNS, which is herein incorporated by reference in its entirety.

FIG. 11shows different performance metrics contained in a value metric table76. There can any number of tables each associated with a different type of performance metric. For example, table76contains latency performance metrics for each storage volume. Other tables77and79include number of read operations per second (I/O Ops) and number of blocks read per second (throughput) metrics, respectively. The metric tables are used by the enforcement logic70and allocation logic80and determine how much tiering media50to allocate to disks that violate their SLA parameters in table75.

In the example shown inFIG. 11, the parameters in column76B predict that every Gigabyte (GB) of DRAM allocated to disk A will approximately reduce the average latency of each read operation associated with disk A by 0.5 msec, reduce the average latency of each read operation associated with disk B by 0.8 msec, and have no effect on disk N. The parameters in column76C indicate that allocating an additional 1 GB of Flash memory to disk A will reduce the average latency of each read operation associated with disk A by approximately 0.5 msec, reduce the average latency of each read operation associated with disk B by approximately 0.3 msec, and have no latency effect on disk N.

The metrics in table76are obtained either from empirical data monitored for previous accesses to these particular disks or are dynamically created and periodically updated based on the prior monitoring of the latency, I/O, and throughput performance of the different disks in table65ofFIG. 4. This will be described in more detail below inFIGS. 12 and 13.

Table76indicates that both DRAM22and Flash20provide that same levels of latency improvement for disk A. In this situation, the allocation logic80may assign Flash20to disk A whenever there is a SLA violation. This may be the case when there is more Flash available than DRAM and/or when the DRAM22is faster than the Flash media20. In other words, whenever different tiering media50provide similar performance improvements, the allocation logic80may allocate the slower and/or more plentiful type of tiering media to the violating storage volume.

The metric values in table76also indicates that allocation of 1 GB of additional DRAM to disk B provides a 0.8 msec reduction in the average read latency where Flash20only provides a 0.3 msec improvement. Accordingly, the allocation logic80may tend to allocate more DRAM22to disk B since DRAM22is more efficient at reducing the latency of read operations from disk B.

The metric values in table76also indicate that allocating additional DRAM22or Flash20to disk N will not further reduce the latency of read operations from disk N. This may be the case when read accesses to disk N are always being performed from different random areas/blocks in disk N. In this situation, temporarily storing portions of the data from disk N into tiering media50may not provide any reduction in read latency since each new read operation will still have to access disk N in storage array24(FIG. 1). Therefore, the allocation logic80will not allocate any more Flash20or DRAM22to disk N, even though disk N may currently be in violation of an associated SLA.

FIG. 12shows how the parameters in tables65,75,76, and85are used by the enforcement logic70and allocation logic80for allocating tiering media. In operation180the enforcement logic70may identify the current average latency for disk A recorded in table65ofFIG. 4as 2 msec. In operation182the enforcement logic70uses table75inFIG. 5to determine that disk A is the biggest SLA violator. In this example, the current 2.0 msec latency for disk A is furthest away from the associated target latency of 0.5 msec indicated in column75D of table75. Similar analysis can be performed for the target number of read operations in column75B of table75and/or for the target block throughput parameters in column75C of table75.

In one embodiment, different target values in table75may have different priorities. For example, a violation of target latency in column75D may have a higher priority than a violation of the target read operations in column75B or the target throughput in column75C. In this case, the tiering media would first be allocated to the disks that are most severely violating the target latency SLA. Otherwise the different SLA parameters in table75may be combined to determine which disks are the worst violators of the SLAs.

In operation184the allocation logic70uses tables76,77, and78inFIG. 11to determine what type and how much tiering media to assign to disk A. For example, table76indicates that Flash20and DRAM22are equally effective in reducing read operation latency. The allocation logic in operation186then uses the metric in column76C for disk A to determine how much additional Flash media20to allocate to disk A.

In this example, disk A has a latency metric of 0.5 msec per GB of Flash20. As previously determined in operation180, the current average read latency for disk A is 2.0 msec. The allocation logic80divides the amount of SLA violation (2 msec=0.5 msec=1.5 msec) by the Flash latency metric (0.5 msec/GB) and determines that an additional 3 GB of Flash20should be allocated to disk A.

The allocation logic80may also compare the maximum resource value in table85inFIG. 9to confirm the identified additional amount of Flash 3 GB does not exceed the maximum allowable allocated resources in column85C (2000 GB) for disk A. If the additional tiering media does not exceed the maximum, an additional 3 GB of Flash memory20is allocated to disk A in operation186and the resources identified as currently allocated to disk A in column85B of table85is increased by 3 GBs.

FIG. 13shows how the metric values in table76are derived by the enforcement logic70. As described above, the resource allocation logic80records the amount of tiering resources currently being used by a particular disk in table85. Allocation logic80also records in column85D of table85the amount of additional tiering media allocated or deallocated from a particular disk during a last SLA enforcement cycle. For example, table85indicates that an additional 10 GBs of tiering media were allocated to disk A during the last SLA enforcement cycle.

The performance monitor60in table65ofFIG. 4also records changes in the average latency for particular disks since a last SLA enforcement cycle. For example, table65indicates a change in the average latency for disk A from time period1to time period2increased by 2 msec.

In operation190, the enforcement logic70uses table85to identify the amount of tiering resources allocated or deallocated to or from particular disks during the last SLA enforcement cycle. In operation192the enforcement logic70uses column65F in table65to determine the change in average latency for particular disks since the last SLA enforcement cycle. In operation194the metrics for table76are derived by dividing the change in average read latency by the amount of last allocated memory. For example, 2 GBs of Flash20may have been allocated to disk A during a last SLA enforcement cycle and the average latency for disk A may have been reduced by 0.5 msec. Therefore, the Flash tiering metric latency in column76C ofFIG. 11for disk A may be updated in operation196to 0.5 msec/2.0 GB=0.25 msec/GB.

The metric values in table76may be determined based on averaging the amount of allocated tiering media and averaging the associated changes in latency, read operations, and block for multiple different SLA enforcement cycles. This reduces large fluctuations in tiering media allocations. Alternatively, the parameters used in table76may be normalized.

It should also be noted that the allocation logic80may also deallocate tiering media from particular storage volumes. For example, value metric table76may include zero and negative numbers indicating that the previous tiering media allocation provided no performance improvement or made the performance of the storage volume worse. The allocation logic80may then deallocate some of the tiering media from the particular disks that have little, none, or negative performance improvements when tiering media is allocated.

The virtualization controller16inFIG. 1more efficiently and effectively utilizes tiering media resource50. For example, tiering media50will not be assigned to storage volumes when allocating additional tiering media does not result in any improvements in storage access performance. Further, when tiering media50is determined to increase storage access performance, the virtualization controller16further increases tiering media efficiency by determining which of the different types of Flash20and DRAM22provides the best performance results for different storage volumes.

Alternative Embodiments

FIG. 14shows an alternative embodiment of the performance virtualization appliance14that performs other SLA enforcement operations. An SLA comparator30measures the IOPs (I), Latency (L), and Throughput (T) parameters32and34for the DRAM22and Flash20, respectively. The SLA comparator30also measures the I, L, and T parameters36for the disk storage array24and measures the I, L, and T parameters38seen by the server10.

These parameters can be determined by the SLA comparator30by monitoring read and write commands to the different memory devices20,22, and24and then determining when a corresponding response is returned by that memory device. For example, a write command to volume26A may be sent from one of the clients10to DRAM22. The SLA comparator30detects the write command to DRAM22and then determines when a corresponding write acknowledge is generated by DRAM22. Similarly, when a read command is detected, the SLA comparator20can determine when the memory device responds back with the data addressed by the read command.

It should be noted that the I, L, and T values for the different memory devices20,22, and24can vary depending on how much memory space is allocated, where the data for a particular volume is located, what types of read and write operations are requested, how much data is associated with the memory access, etc. Accordingly, monitoring the I, L, and T values in real-time allows the SLA comparator30to constantly monitor and determine if the overall SLA value is being provided for a particular volume26.

A combiner44combines the performance values from DRAM22, Flash20, and disk storage array24at any given instant to determine if the PVA14is providing the overall SLA value requested by the client10. For example, the average values for I, L, and T for each memory device20,22, and24may be combined pro-rata according to the amount of data accesses for each memory device over a particular period of time.

For example, if 30% of the memory accesses for a particular volume26are from DRAM22, then the average DRAM access time for a particular time period may be given 30% of the overall weighting for the overall latency value. The average access times for Flash20and disk storage array24are given similar weights according to the percentage of accesses for that same time period. The combined I, L, and T values38are then used to determine if the SLA value for a particular volume is currently being provided.

The SLA comparator30outputs K parameters40according to the identified I, L, T values32,34,36, and38. Proportional controllers42A,42B, and42C use the K parameters K1, K2, and K3, respectively, from the SLA comparator30to determine how much RAM22and Flash20should be used at any given instant.

Assume that the DRAM22has a latency value (L) of 1 micro-second (μS). For example, a read and/or write from DRAM22takes 1 μs before the data is available/acknowledged. Also assume that Flash20has a 250 μS latency for read/write, and the disk storage array24has a read/write latency of 5 milli-seconds (5,000 μS).

Also assume that the volume26A is 100 Giga-Bytes (GB) and a particular user has requested an SLA of 100 μS latency (L) for the volume26A. The PVA14uses a measurement and operation interval of T measured in time units. For example, the time interval T may be 60 seconds. The overall measured value of L at the output of the combiner44is referred to as Z and is the latency for volume26A seen by the client10.

The SLA comparator14strives to maintain a particular ratio of K1and K2. K1is the coefficient output from the SLA comparator30to the proportional controller42A for RAM22. K2is the coefficient output from the SLA comparator30to the proportional controller42B for Flash20. A value K3=1−(K1+K2) is the coefficient output from the SLA comparator30to the proportional controller42C for the disk storage array24.

The SLA comparator30generates K1and K2values so that the overall latency Z=100 μS (where Z=K1*1+K2*250+K3*5000). For example, if the overall value of Z is slower than 100 μS, the SLA comparator30may increase the value of K1and reduce the value of K2and/or K3. This has the overall effect of reducing the overall latency Z.

The K1, K2, and K3values40are used by the proportional controllers42A,42B, and42C, respectively, to vary the amounts of DRAM22, Flash20, and disk storage array24used for a particular volume26A. For example, a particular amount of DRAM22may be associated with the current K1value used by controller42A. If the SLA comparator30increases the current K1value, the controller42A may load a larger amount of volume26A into DRAM22.

If the overall latency Z is less than the SLA1, then the comparator30may decrease the K1value and/or increase the K2value. This may cause the controller42A to de-allocate some of the DRAM22for volume26A and/or cause controller42B to allocate more Flash20to volume26A.

For reads from the client10, relevant data is pre-loaded, pre-fetched, or continuously cached in DRAM22and Flash20by the proportional controllers42according to the K values40. For writes, the controllers42A and42B accept a certain amount of write data into DRAM22and Flash20, respectively, according to the K values40and later write the stored write data back to the disk storage array24.

If a K1and/or K2value is dynamically increased, the corresponding controller42A and/or42B either pre-fetches more of the corresponding volume26from disk storage array24into DRAM22and/or Flash20; or writes more data for that volume from the clients10into DRAM22and/or Flash20.

If a K1and/or K2value is dynamically decreased for a particular volume26, the corresponding controller42A and/or42B either down-loads more of that corresponding volume from DRAM22and/or Flash20back into disk storage array24or directs more write data for that volume from clients10into disk storage array24.

The system described above can use dedicated processor systems, micro controllers, programmable logic devices, or microprocessors that perform some or all of the operations. Some of the operations described above may be implemented in software and other operations may be implemented in hardware.

Having described and illustrated the principles of the invention in a preferred embodiment thereof, it should be apparent that the invention may be modified in arrangement and detail without departing from such principles. We/I claim all modifications and variation coming within the spirit and scope of the following claims.