Handling path issues for storage copy services

A method for determining path health to conduct a plurality of Input-Output (IO) operations along a healthy path in a network is provided. The present invention may include receiving an IO request from a user and sending the received IO request on a first path. The present invention may include determining a first IO response has exceeded a threshold time on the first path. The present invention may include determining the first path has degraded based on the exceeded threshold time. The present invention may include generating a duplicate IO request and sending on a second path. The present invention may include receiving the duplicated IO response before receiving the original IO response. The present invention may include determining a health state associated with the slower path. The present invention may include refreshing a path state machine based on the determined health state associated with the slow path.

BACKGROUND

The present invention relates generally to the field of computing, and more particularly to storage copy services.

Generally, data is mirrored to another storage location over networks for data to be quickly recovered in the event of interruption. A mirror site replicates the original site and is typically updated frequently to match the original site. Mirror sites can also allow for faster access to data from a geographically distant location. For instance, if an original site is located thousands of miles away from the customer accessing the data, a mirrored site closer in proximity could offer the same data at a faster speed.

SUMMARY

Embodiments of the present invention disclose a method, computer system, and a computer program product for determining path health to conduct a plurality of Input-Output (IO) operations along a healthy path in a network. The present invention may include receiving an original IO request from a user. The present invention may also include sending the received original IO request on a first path from a primary site to a secondary site. The present invention may then include determining a first IO response associated with the sent original IO request has exceeded a threshold time on the first path between the primary site and the secondary site. The present invention may further include determining the first path has degraded between the primary site and the secondary site based on determining that the first IO response has exceeded the threshold time. The present invention may also include generating a duplicate IO request based on the received original IO request. The present invention may then include sending the generated duplicate IO request on a second path. The present invention may further include receiving a second IO response associated with the sent duplicate IO request on the second path. The present invention may also include receiving the first IO response associated with the original IO request on the first path. The present invention may then determine that the first IO response was received on the first path after the second IO response was received on the second path. The present invention may further include determining a health state associated with the first path based on determining that the first IO response was received on the first path after the second IO response was received on the second path. The present invention may also include refreshing a path state machine based on the determined health state associated with the first path.

DETAILED DESCRIPTION

The following described exemplary embodiments provide a system, method and program product relating to path health response processes. As such, the present embodiment has the capacity to improve the technical field of storage copy services by providing a method that measures the health state of a path by comparing the latency between paths to determine if a path has degraded. A path state machine corresponding to each path may be used to indicate the state of a path. More specifically, a latency calculation may be used to measure the latency on a path for the same uniquely numbered IO request sent along two different paths between the primary storage device and the secondary storage device by comparing the degraded path with another good path. Finding latency on a degraded path may allow a slower path to be avoided by sending the duplicate mirrored IO request using a healthy faster path. By detecting a problem path between a primary and secondary site and duplicating the IO operation to send on a faster path, the data storage system may perform optimally and allow customers to not be impacted by a slow path.

As described previously, generally data is mirrored to another storage location over networks for data to be quickly recovered in the event of interruption. A mirror site replicates the original site and is typically updated frequently to match the original site. Mirror sites can also allow for faster access to data from a geographically distant location. For instance, if an original site is located thousands of miles away from the customer accessing the data, a mirrored site closer in proximity could offer the same data at a faster speed.

Currently, if there is a problem on the network between the primary site and the secondary site, the performance of the whole system may deteriorate. A current typical method for detecting path failure may be to use a threshold to identify the health state of one path and when the path's health state exceeds the threshold, the path may be suspended. During this suspended period, the workload may not be cut off, rather the workload may be reduced on that path while the path is still being used. This may create an inability for the local site to detect whether the slowness is a path issue or if the slowness is due to bad performance of the remote site's storage system.

When a path has exceeded the threshold and the path is in a degraded state, the data may still be transferred on this path and during this period the performance impact may be apparent to the customer. Further, once the path is set to a degraded state, there may be no method to recover the path automatically. Although the path slowness may be caused by a transient event (e.g., adapter reset or code upgrade), the path may not automatically recover once the path has failed. Therefore, it may be advantageous to, among other things, provide a way to automate path recovery when a path has degraded to ensure optimal performance over a multi-path network.

According to at least one embodiment, recovery action may be taken at the first instance of a path failure which can minimize the impact of a degraded IO path to the customer. The health state of one path may be measured by using an Exponential Moving Average (EMA) calculation which can indicate the relative state of path health between one path and all other paths in the system. An EMA comparison may be based on the latency on a path for the same IO, which can eliminate the impact of other components in the system, such as backend drive impact or time difference impact.

A path state machine may be used to identify the path health of a path to keep optimal performance on a network when one path becomes degraded. When a path becomes degraded, a duplicate IO request (e.g., a write request) may be sent using another good path while the original IO request is on the degraded slower path. Thus, from an application level, there may be no performance impact when one path is degraded. Data consistency may not be harmed on the secondary site if the same IO request is repeated more than one time. Each IO request may have a unique sequence number for identification, therefore, on the primary site, when the second IO response comes back, the primary site drops the IO response without harm since the IO operation had already been performed.

Performance may be maintained when a path fails by taking action at the first indication of failure by duplicating and sending the unique sequence numbered IO request using another good path. The health state along the path may be measured using the latency responses of the same IO requests sent along two different paths. Detecting and correcting degraded paths may eliminate the impact to other components (e.g., backend Redundant Array of Inexpensive Disks (RAID) overload for some specific IO). Additionally, paths may be automatically recovered after a transient error.

The system threshold path states can be represented by using a path state machine for each path. The path state machine can show whether the path is in a good, degraded or suspended state. The path state may be detected based on system threshold differences such as when the path times-out and through latency calculations. Time-out can occur when the latency of the application for one IO operation exceeds a threshold period (e.g., 2 seconds for an Online Transaction Processing (OLTP) application). Thus, the 2 second latency can show that the path is performing poorly and can be put into a degraded path state. Thereafter, another IO path may be chosen for the duplicate IO operation, allowing the degraded path to be bypassed by sending the IO request using another path.

A path may transition from a degraded state to a good state when an IO response on a path has exceed a predefined percentage (e.g., 95%) within a time window (e.g., 3 minutes) and when no time-out occurred on the path for a specific period of time (e.g., 5 minutes). The latency recovery for the path state machine state to transition from a degraded state to a good state may be calculated using the Exponential Moving Average (EMA) value for latency. Latency recovery may occur, for example, when the EMA latency value drops to near 100% (e.g., 110%) and at least a certain number of IO operations are monitored (e.g., 500 IO operations), then the path can be recovered to a good state. A 100% EMA value for a number of continuous IO request operations can show that the path has nearly the same latency compared to other paths, thus the path may be determined to be in a good state. Upon determining another IO operation time-out has occurred, the EMA calculation may be used again to determine path state.

The current health state indicated by a path state machine may transition from a degraded state to a suspended state when the EMA value has been above a threshold (e.g., 200%) for a period of time. Once a path is in a suspended state, physical repair (e.g., a reboot) may be required for the path to transition back to a good state.

The EMA calculation value may assist in checking the state of a path. When determining the EMA value to check whether a path state has transitioned from a degraded state to a good state, a latency comparison may be used to compare the latency of the degraded path with the latency of another good path for the same IO request. The latency comparison may be referred to as Latency Index, where one such calculation can be shown by the following:
LatencyIndex=latency_on_degrade_path/latency_on_good_path.

A decision window may monitor the number of IO operation requests (e.g., N requests) to show the path state stability for a period of time. Finally, the following EMA calculation may be used to calculate an EMA value to avoid spike issues:
EMAnew=α*LatencyIndexnew+(1−α)*EMAnew
where α=2/(N+1) and N may be the count of IO operations that may be monitored before determining the adjustment of the path state.

When all the paths have the same performance or health state, the EMA may be 100%, however, a reasonable real world deviation may define the threshold EMA to be approximately 110%, allowing a tolerance of approximately 10% performance deviation among the paths. The EMA calculation may also be considered for measuring when a path is not functional and may not be recovered. The threshold EMA may be set at a higher value than 100% (e.g., 200% which is two times worse) for a specified number of continuous IO requests.

According to the present embodiment, a user using a client computer102or a server computer112may use the path recovery program110a,110b(respectively) to send and receive IO operations timely in the event of path failure. The path recovery program110a,110bhas the capability of determining the path health to ensure the IO operations are not slowed down even if one path is not performing optimally. The path recovery method is explained in more detail below with respect toFIGS. 2-4.

Referring now toFIG. 2, a system block diagram illustrating a system network200according to at least one embodiment is depicted. The system network200may include the client device202(e.g., client computer102) capable of sending and receiving IO operations, a primary storage device204and a secondary storage device206.

The client device202may transmit an IO operation to the primary storage device204via a communication network116. The primary storage device204(e.g., server112) may include a path recovery program (e.g.,110b). The primary storage device204may send IO operations to and receive IO operations from the secondary storage device206(e.g., server112). The primary storage device204may also be connected to a secondary storage device206via a communication network116using multiple IO paths208a-208cwithin the system network200.

The secondary storage device206may act as a mirror to the primary storage device204by performing the same IO operations on identical data as the primary storage device204. The secondary storage device206may be used as a data mirror in a storage environment to achieve high availability of data to a customer. A data mirror may be used between two different storage systems, such as primary storage device204and secondary storage device206, in different locations. In order to keep the data consistent between the primary storage device204and the secondary storage device206, the mirrored system may be synchronized. For example, when writing data on a synchronized system, the operation may be complete on both systems before the customer can acknowledge the data. Since the distance between the primary device204and the secondary device206may be far, the distance between the primary storage device204and the secondary storage device206may contribute to the latency of the network and the response time to the customer.

The path recovery program (e.g.,110b) running on the primary storage device204may calculate and analyze which path208a-cthe IO request should use to increase efficiency for the client using the client device202. The path recovery program (e.g.,110b) may duplicate an IO request since the original IO request may have been sent on a default path (e.g.,208a) that has degraded. The same unique sequence numbered duplicate IO request may be sent along another good path (e.g.,208b) to ensure the client device gets a response that is not delayed due to the degraded default path (e.g.,208a).

Referring now toFIGS. 3A and 3B, an operational flowchart illustrating a degraded path health response process300used by the path recovery program110a,110baccording to at least one embodiment is depicted.

At302, the path recovery program110a,110bgenerates a unique sequence number. Generating a unique sequence number may be done using known methods such as a random non-repeating sequence number generator or a chronological non-repeating sequence number generator. A unique sequence number may be assigned to each IO request (e.g., write request) and may be used to determine if two IO requests are identical. An original IO request may utilize the default path208awhich may be functioning properly. Alternatively, a duplicated IO request may be used if a time-out on a default path208ahas occurred. The path recovery program110a,110bmay utilize another good path208b, other than the default path208a, to send the IO request upon duplication. A duplicated IO request may have the same unique sequence number as the original IO request to indicate if the IO request has already been processed, therefore, the duplicated IO requests may not be processed twice. A unique IO sequence number may also be used to test the health state of an IO path (e.g.,208a-c). By creating a unique sequence number, the path recovery program110a,110bmay use a mirror operation to send a duplicate IO request through a different path208a-cif one path208a-cbecomes inoperable. For example, when a user enters a write IO request, the path recovery program110a,110bwill generate unique sequence number 125773 and assign that unique sequence number 125773 to that specific write IO request.

Next, at304, the path recovery program110a,110bwill send the IO request to a remote site (e.g.,206) with this unique sequence number using a default path208a. The path recovery program110a,110bmay send the IO request from the primary storage device204to the secondary storage device206using the default path208athat was previously in use. Additionally, the workload may be distributed evenly along the paths208a-cin the system network200. For example, the IO request with a unique sequence number will be sent to the mirrored secondary storage device206using the default path208athat had been in use previously. In the case of multiple IO requests, the workload may be evenly distributed until one path208a-cdegrades, causing that IO request to slow down on the degraded path.

Then at306, the path recovery program110a,110bdetermines if the default path208ahas degraded by not receiving a response within a preset threshold time. A path208a-cthat has degraded may indicate the path208a-cis not functional enough to provide a normal IO operation without delaying the IO operation. To determine whether a path208a-chas degraded, the path recovery program110a,110bmay recognize latency on the default path208aby a time-out where the threshold time has exceeded a set time period. Action may be taken at the first instance of a time-out to respond to a degraded path state. Taking action at the first time-out may allow the degraded path208ato recover faster since the path recovery program110a,110bmay begin to immediately use another good path208bfor IO operations. For example, if the preset threshold time is set at 2 seconds and no response is received within the 2 second threshold time, then the path208aexceeds the threshold value and the path208ais considered to be in a degraded state.

If the path recovery program110a,110bdetermined that the path208a-chas degraded at306, then the path recovery program110a,110bwill randomly choose another good path208band repeat the IO operation with the same sequence number at308. Once a degraded path208ahas been identified, the path recovery program110a,110bmay begin duplicating IO operations and comparing latency response times between a good path (e.g.,208b) and a bad path (e.g.,208a). A duplicate IO operation may originate from many different hardware and software components, however, the same hardware and software components may be used in a duplicate IO operation for optimal latency calculations. If the hardware components or software components change from one IO operation to the next, that change may create different latencies within the network system which may affect the time-out operation. The path recovery program110a,110bmay change the default path208abeing used for the duplicate IO operation instead of changing other hardware or software components. For example, if two different paths208a,208bare used for the same IO operation when the default path208ahas been degraded, then sending the duplicate unique IO request using another good path208bwill create an optimal IO response time for the user. The backend may involve the same disk or the same controller so difference in response time for the IO operation may be accurately calculated. The difference may be determined by comparing the response time (i.e., latency) of an alternate good path208bwith the response time of the degraded path208a.

If the path recovery program110a,110bdetermined that the path has not degraded at306, or if the path recovery program110a,110brandomly chose another good path208band repeated the IO request with the same sequence number at308, then the path recovery program110a,110bwill wait for the IO response at310. The IO response received may correspond to the original IO request if the path208awas not degraded or the duplicated IO request that was sent on another good path208bif the default path208awas determined to be degraded.

At312, the path recovery program110a,110bdetermines if the response is a first received response associated with the unique sequence numbered IO. The IO response received may correspond to the original IO request or may correspond to a duplicated IO request. The response time for each IO response may depend on the health of the path208a-c. The path recovery program110a,110bmay determine that an IO response corresponding to the unique sequence number is based on the sequence number of the response matching the unique sequence number. A first IO request and corresponding response may be determined by the path recovery program110,110bto be along the default path208athat may be transmitting the IO operation normally (i.e., not in a degraded state). Alternatively, the duplicate IO request and corresponding response may be determined by the path recovery program110a,110bto be along another good path208bif the default path208ais in a degraded state. Subsequent IO responses having the unique sequence number (e.g., any IO response that is not the first IO response) received by the path recovery program110a,110bmay not correspond to the first IO request and may result in a latency calculation. Receiving a subsequent unique sequence numbered duplicate IO response may indicate the default path208amay have been degraded and the path recovery program110a,110bduplicated the IO request having the same sequence number as the original IO request. Further, the path recovery program110a,110bmay have sent the duplicated IO request along another good path208b, making the duplicated IO response the first IO response to arrive for the unique sequence number.

Continuing the example used previously, a user's write IO request with unique sequence number 125773 may be transmitted on the original default path208ato the secondary storage device206when the path208amay be in a good state. Thereafter, the secondary storage device206may perform the IO operation and generate and send an IO response having a sequence number 125773. Therefore, IO response 125773 may be the first and potentially only IO response to be received as no duplicate may be necessary if the path208ais fully functional. If the path state of the original default path208ais in a degraded state, the user's write IO request 125773 may be duplicated with the same sequence number (i.e., 125773) allowing the user's write IO request to be transferred through another good path208bwithout any latency from the degraded path208a. Therefore, the response corresponding to the duplicated write IO 125773 may be the first IO response to be received whereas the response corresponding to the original IO request 125773 may follow from the degraded path208aand arrive later due to greater latency.

If the path recovery program110a,110bdetermined that this is a first response for the sequence number at312, then the path recovery program110a,110bwill acknowledge the customer IO is complete at314. A first IO response from the default path208amay be acknowledged by the path recovery program110a,110bindicating that the customer IO operation may be complete if the IO response is the first IO response received for the sequence number. Also, a duplicated IO response from the other good path208bmay be acknowledged by the path recovery program110a,110bindicating that the customer IO may be complete if the duplicated IO response is the first IO response received for the sequence number. For example, if a user initiates a write IO request and the path recovery program110a,110bsends the IO request on a healthy default path208a, then the first IO response was handled and acknowledged as complete without any duplicate IO requests. Alternatively, if a user initiates a write IO request and the path recovery program110a,110bsends the IO request along a default path208athat is in a degraded state, then the duplicated IO request will be generated and sent along a good path208b, making the duplicated IO response the first response to be acknowledged and completed. The customer may receive the first acknowledgement whether the first IO response was derived from the first IO request or if the first IO response was derived from the duplicated IO request.

Then at316, the path recovery program110a,110bdetermines if the response corresponding to the first IO request is detected. A detected IO response may be determined by which path208a-cthe IO request and corresponding response uses. The path recovery program110a,110bmay detect an IO response that may be from another good path208band may not detect an IO response from the default path208a. An undetected IO response received from a default path208acan indicate the IO response may be complete and the default path208awas functional for the first IO request. An undetected IO response ends the path recovery program110a,110b. For example, if a user initiates a write IO request when the default path208ais fully functional, then the IO request utilized the default path208afor the IO operation and the customer IO request may be complete, ending the path recovery program110a,110b. Alternatively, a write IO request sent from the user when the default path208ais in a degraded state may cause the write IO request to be duplicated and sent along another good path208b. Further, the degraded default path208amay cause the duplicated IO response to be the first response received due to the original write IO request being slowed down on the default path208a. Then the path recovery program110a,110bmay acknowledge the IO response corresponding to the duplicate IO request sent on the other good path208bas complete for the user and the duplicated IO response may be detected since the IO response used the other good path208b. IO detection relates to which path208a-cthe IO request and corresponding response uses. Therefore, an IO response may be detected if the IO request was sent on another good path208band the IO response may not be detected if the IO request was sent on the default path208a.

If the path recovery program110a,110bdetermined that the IO response is detected at316, then the latency for the first IO response will be recorded at318. The latency may be recorded in a data repository, such as a database114, for an IO response that arrived on another good path208b(i.e., the duplicated IO response). The IO response detection that occurred at316signals the path recovery program110a,110bto record the latency of path208band return to310to wait for another response from the default path208aas the default path208awas slower than another good path208bfor the first IO response. The health state of the default path208aindicates the functionality of the default path208aby measuring the latency with the same IO operation. For example, if the user sends a write IO request and the path recovery program110a,110bdetermines that the default path208ahas degraded, then the path recovery program110a,110bwill send a duplicated IO request along another good path208b, which may be subsequently detected as described previously at316. The path recovery program110a,110bwill then record the latency (e.g., a value indicating the difference between a time stamp indicating when the request is sent versus another time stamp indicating when the response is received) in a database114and then return to310to wait for another IO response.

If the path recovery program110a,110bdetermined at312that the received response does not correspond to the first response for the sequence number, then the path recovery program110a,110bwill compare the latency of the degraded path208awith the latency of another good path208bat320. Comparing the latency between the degraded path208awith the latency of another good path208bmay show a time difference where the degraded path208acorresponds to a slower IO response received and the other good path208bcorresponds to the faster IO response received. The original IO response may correspond to the IO request sent using the degraded path208awhile the duplicated IO response may correspond to the IO request sent using another good path208b. Therefore, the original IO response may have a slower response time for the unique sequence number while the duplicate IO response may have a faster response time for the unique sequence number.

The difference between the latency of the degraded path208aand the latency of the other good path208bmay be calculated each time the unique sequence numbered IO response is received and subsequent latency recalculations may be made until the degraded path208ais determined to no longer be in a degraded state. The latency associated with each degraded path208amay be compared to the latency of the same unique sequence IO response sent along another good path208bthat was previously detected. For example, a user who sends a write IO request through the path recovery program110a,110bwill receive a complete acknowledged response regardless if the default path was fully functional or degraded due to the path recovery program's110a,110bability to respond by sending the IO request through another good path208bthat may be fully functional. The path recovery program110a,110bmay make subsequent latency calculations between the latency associated with another good path208band the latency associated with a degraded path208ato determine when the degraded path208amay no longer be in a degraded state, further allowing the path recovery program110a,110bto begin utilizing the default path208aonce the default path208amay no longer be in a degraded state.

At322, the path recovery program110a,110bcalculates the EMA value between two paths208a-c. The health state of one path208a-cmay be measured by using an Exponential Moving Average (EMA) calculation which may indicate the relative state of path health between one path (e.g.,208a) and all other paths (e.g.,208band208c) in the system. An EMA threshold calculation may be based on comparing the recorded latency from the detected IO response on another good path208bpreviously described at318and the second IO response on the degraded path208a. When all the paths have the same performance or health state, the EMA may be 100%, however, a reasonable real world deviation may define the threshold EMA to be approximately 110%, allowing a tolerance of approximately 10% performance deviation among the paths.

Then at324, the path recovery program110a,110bwill refresh the path state machine. The refresh is based on the EMA threshold value. The path state machine may be refreshed to reflect the current health state of the path208a-cas will be described in detail below with reference toFIG. 4.

Referring now toFIG. 4, a state diagram illustrating a path state machine400that represents three health states of a path208a-cbetween the primary storage device204and the secondary storage device206is depicted according to at least one embodiment. The path health states represented within the path state machine400may include a good path state402, a degraded path state404and a suspended path state406.

Each instance of the path state machine400may be handled by the path recovery program110a,110bon the primary storage device204. Each path208a-cmay have a corresponding instance of a path state machine400where the default state may be a good path state402. A good path state402may indicate that the corresponding path208a-cmay be fully functional. A degraded path state404may indicate that the corresponding path208a-cmay not provide sufficient performance for normal IO operations. A suspended path state406may indicate that the corresponding path208a-cmay be in a suspended state where the path208a-chas failed and needs repair before reuse since the IO operation may not reach the targeted destination. The path state may be detected based on system threshold differences such as when the path208a-ctimes-out and through latency calculations.

The path recovery program110a,110bmay transition the current path state of a path208a-cfrom a good state402to a degraded state404once an IO operation time-out has occurred. Time-out can occur when the latency of one IO operation exceeds a threshold time (e.g., 2 seconds for an Online Transaction Processing (OLTP) application). Thus, the path recovery program110a,110bmay determine that a path208a-cthat has a 2 second latency exceeds the time-out threshold and is performing poorly, thus the current path state may transition into a degraded path state404. Thereafter, another IO path208bmay be chosen as described previously at308, that may allow the degraded path208ato be bypassed by sending the duplicate IO request using another good path208b. For example, if a user write IO request is sent along the default path208aand the default path208ais slow enough to exceed a threshold time, then the duplicate write IO response will return to the primary storage device204while the slower write IO response latency time will be used in a calculation by the path recovery program110a,110b. In another example, if the latency calculation for the slower write IO is determined by the path recovery program110a,110bto exceed 2 seconds, then the current path state of the default path208awill transition to a degraded path state404. Subsequently, the degraded path208astate will either transition to a good path state402if the latency is recovered or will transition to a suspended path state406if the IO operation exceeds a threshold.

The path recovery program110a,110bmay transition the current path state of a path208a-cfrom a degraded state404to a suspended state406when the threshold on the degraded state404has been exceeded for a predetermined period of time. Similarly, if the path recovery program110a,110bdetermines the degraded state404cannot be recovered, then the current path state of the path state machine400may transition to a suspended state406. An example of the exceeded threshold may be calculated by the EMA value. The value resulting from the EMA calculation may assist in determining the health state of a path208a-c. When determining the EMA value to check whether a path state threshold has changed, the path recovery program110a,110bmay compare the latency between two paths208a-c. The Latency Index calculation may be shown by
LatencyIndex=latency_on_degrade_path/latency_on_good_path.
For example, once the path recovery program110a,110bhas determined the path208a-cis in a degraded state404for a period of time, such as 3 seconds, then the current path state of the path208a-cwill transition to a suspended path state406. A path208a-cis considered to be suspended once the path recovery program110a,110bcan detect the path208a-cis positioned to fail by exceeding the EMA threshold for a period of time (e.g., 3 seconds) or if physical failure happens on the path208a-c.

From a suspended state406, the path recovery program110a,110bmay transition the current path state of a path208a-cto a good state402by repairing the path208a-c. Repairing a path208a-cfrom a suspended state406to a good state402may consist of replacing the network link or some other reset of the link if a normal recovery action is to manually reset the link.

The path recovery program110a,110bmay transition the current path state of a path208a-cfrom a degraded state404to a good state402by determining the latency has recovered. Determining whether the latency has recovered for the path state machine400to transition the current path state from a degraded state404to a good state402may be calculated by using the Exponential Moving Average (EMA) value for latency. An example of latency recovering could be determined when the EMA latency value drops to near 100% (e.g., 110%) and at least a certain number of IO operations are monitored (e.g., 500 IO operations), then the path208a-cmay be functioning optimally and the current path state may be updated to a good path state402to reflect the current state of the path208a-c. A decision window may be used to monitor the number of IO operation requests (e.g., N requests) to show the path state stability for a period of time. To avoid spike issues when recovering latency, the path recovery program110a,110bmay calculate the EMA as follows:
EMAnew=α*LatencyIndexnew+(1−α)*EMAnow
where α=2/(N+1) and N may be the count of IO operations that may be monitored before determining the adjustment of the path state. A 100% EMA value for a number of continuous IO request operations can show that the path208a-chas nearly the same latency compared to other paths208a-c, thus the path208a-cmay be determined to be in a good state402. Upon determining another IO time-out, the EMA calculation may be used again to determine the current path state. For example, when an IO request on a path208a-chas exceeded a predefined percentage (e.g., 95%) within a time window (e.g., 3 minutes) and when no time-out occurred on the path for a specific period of time (e.g., 5 minutes), the current path state of the path208a-cmay transition from a degraded state404to a good state402.

The path recovery program110a,110bmay transition the current path state of a path208a-cfrom a good state402to a suspended state406if physical failure on the device occurs. For example, if a port is backlogged or if a physical failure is present, then the current path state will transition from a good state402to a suspended state406which will need to be repaired to return to a good state402again.

It may be appreciated thatFIGS. 2-4provide only an illustration of one embodiment and do not imply any limitations with regard to how different embodiments may be implemented. Many modifications to the depicted embodiment(s) may be made based on design and implementation requirements.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Workloads layer1144provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation1146; software development and lifecycle management1148; virtual classroom education delivery1150; data analytics processing1152; transaction processing1154; and path recovery1156. A path recovery program110a,110bprovides a way to send and receive IO operations timely in the event of path failure. The path recovery program110a,110bhas the capability of determining the path health to ensure the IO operations are not slowed down even if one path is not performing optimally.