Context sensitive detection of failing I/O devices

Methods for context sensitive detection of failing I/O devices sample and record a response time of an I/O device for each of a first plurality of time intervals to generate a first plurality of sampled and recorded response times, and to determine whether or not at least one I/O error has occurred in each of the first plurality of time intervals. A mathematical model is applied which characterizes the first plurality of sampled and recorded response times. The mathematical model is applied in accordance with an I/O device category corresponding to the I/O device. The mathematical model provides a frame of reference for defining an I/O failure.

TRADEMARKS

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to the field of computer systems management and, in particular, to methods, systems, and computer program products for providing context sensitive detection of failing I/O devices.

2. Description of Background

Large computing systems typically include a plurality of processor nodes and I/O devices. The nodes are capable of executing an operating system. A subset of these nodes are designated to act as server nodes. The remaining nodes, designated as non-server nodes, may perform input/output (I/O) operations on an I/O device, such as a data storage device or disk drive, through a server node or over a local path. The operating system is provided with a function to detect when an I/O request to a device has not completed within a reasonable amount of time. This approach is problematic because the concept of a “reasonable” amount of time might vary from situation to situation, and the user does not have sufficient information from which to determine an appropriate waiting time. Oftentimes, the actual length of time that a user waits for an I/O device to respond is much too long. For example, the wait may be caused by a I/O device performing its local recovery. If the local device is successful, then the I/O device is usable, but if the recovery is not successful or takes an excess period of time then the I/O device is unable to perform the necessary function. This results in I/O devices which are not functional being left in the configuration longer than is needed. Work is stalled longer than necessary waiting for the I/O request to complete. Accordingly, what is needed is an improved technique for detecting missing I/O interrupts and failures in I/O devices.

SUMMARY OF THE INVENTION

Methods for context sensitive detection of failing I/O devices sample and record a response time of an I/O device for each of a first plurality of time intervals to generate a first plurality of sampled and recorded response times. The sampled and recorded response times are subsequently used to determine whether or not at least one I/O error has occurred in each of the first plurality of time intervals. A mathematical model is applied which characterizes the first plurality of sampled and recorded response times. The mathematical model categorizes the samples by time, transaction volume or other external information to provide highly accurate models which learn from observing the behavior of the I/O devices. The mathematical model is applied in accordance with an I/O device category corresponding to the I/O device. The mathematical model provides a frame of reference for defining an I/O failure. A test is performed to ascertain whether or not at least one I/O error occurred during the first plurality of sampled and recorded response times. When the first plurality of sampled and recorded response times includes at least one time interval during which an I/O error occurred, the response time of the I/O device is sampled and recorded for each of a second plurality of time intervals to generate a second plurality of sampled and recorded response times. The second plurality of sampled and recorded response times is compared to a set of predicted response times generated using the applied mathematical model. If the second plurality of sampled and recorded response times deviates from the set of predicted response times by at least a user defined amount, then the I/O device is switched to an alternate I/O device, or an alert is triggered, or both.

System and computer program products corresponding to the above-summarized methods are also described and claimed herein. Other systems, methods, and/or computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional systems, methods, and/or computer program products be included within this description, be within the scope of the present invention, and be protected by the accompanying claims.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1is a block diagram setting forth an illustrative operational environment in which the present invention is employed. In particular, a plurality of nodes are interconnected over a network104. A subset of these nodes are designated to act as server nodes100.1,100.2. Although the example ofFIG. 1shows two server nodes100.1,100.2, any number of one or more server nodes may be provided. The remaining nodes, designated as non-server nodes100.3through100.n, perform input/output (I/O) operations on a storage device through a server node or over a local path. Nodes100.1through100.n are operably coupled to network104through one or more adapters, cables, switches, or any of various combinations thereof.

In preferred embodiments of the present invention, each node100.i is a processor node capable of communicating with other processor nodes using the publicly defined Transmission Control Protocol/Internet Protocol (TCP/IP) messaging protocol or FIBER Channel or FICON. While this protocol is referred to as an Internet Protocol, it should be noted that use of this term herein does not imply the existence of any Internet connection, nor does it imply dependence upon the Internet in any way. It is simply the name of a conveniently used, well characterized communication protocol suitable for use within a connected network of data processing nodes.

Each node100.i may include one or more Central Processing Units (CPUs), some or all of which share memory with one another. One or more of these CPUs are capable of implementing an operating system. Each node100.i may be connected locally to a non-volatile storage device such as a Direct Access Storage Device (DASD) unit or other similar storage device200.i, where i is an integer greater than or equal to 1, but less than or equal to n. Storage device200.i typically comprises a rotating magnetic disk storage unit, sometimes referred to as a disk drive. However, the scope of the present invention includes any nonvolatile storage mechanism capable of holding data files. The number n of nodes100.i is not critical. Furthermore, not everything operably coupled to network104has to be a data processing node. A plurality of DASD storage devices300.1through300.m are connected to network104using, for example, a network adapter300for maintaining communication between DASD storage devices300.1to300.m and network104.

The system ofFIG. 1includes one or more sensing mechanisms for sampling a response time of an I/O device for each of a first plurality of time intervals to generate a first plurality of sampled response times. These one or more sensing mechanisms may be embedded within the operating system of a server node100.i, or embedded within a storage device200.i, or embedded within a storage area network (SAN) manager405, or embedded within various combinations of the foregoing elements. For illustrative purposes, the configuration ofFIG. 1shows a sensor402embedded in storage device200.2, a sensor400embedded in SAN manager405, and a sensor401embedded in server node100.1. However, it is not required for sensors to be embedded in each of these elements, so long as the configuration ofFIG. 1includes at least one sensor.

Sensors400,401,402sample the response times of one or more I/O devices such as DASD storage devices300.1to300.m. These response times are stored in a storage mechanism, operatively coupled to the sensors400,401,402. For example, information sampled by sensor401may be stored at server node100.1, information sampled by sensor400may be stored at SAN manager405, and information sampled by sensor402may be stored at storage device200.i. The stored information is accessed by a processing mechanism, such as server node100.1, or server node100.2, or SAN manager405, or any of various combinations thereof. The processing mechanism uses the first plurality of sampled response times to determine whether or not at least one I/O error has occurred in each of a first plurality of time intervals, as will be described in greater detail with reference toFIG. 2.

FIG. 2is a flowchart setting forth an illustrative operational sequence for providing context sensitive detection of failing I/O devices. The procedure commences at block201where a response time of an I/O device is sampled and recorded for each of a first plurality of time intervals to generate a first plurality of sampled and recorded response times. The results of block201are used later to determine whether or not an error has occurred. For purposes of illustration, an I/O error may include one or more missing interrupts. Next, at block203, a mathematical model is constructed or selected which characterizes the first plurality of sampled and recorded response times. The mathematical model is constructed or selected in accordance with an I/O device category corresponding to the I/O device. For example, a probability model using binomial distribution may be appropriate in situations where the response time is stable over time. If the response time varies over time, then a model based upon a categorical analysis regression tree may be appropriate. The acceptable behavior of the I/O device will determine which mathematical model is constructed or selected. Essentially, the mathematical model provides a context or a frame of reference for defining an I/O failure. A test is performed to ascertain whether or not at least one I/O error occurred during the first plurality of sampled and recorded response times (block205). If not, the program loops back to block201, as the purpose here is to characterize the response-time distribution in the presence of an error. Errors can be detected with reference to an observed response-time distribution that exists in the presence of an error. Errors can also be detected via missing interrupts.

The affirmative branch from block205leads to block207where the response time of the I/O device is sampled and recorded for each of a second plurality of time intervals to generate a second plurality of sampled and recorded response times. At block209, the second plurality of sampled and recorded response times is compared to a set of predicted response times generated using the constructed mathematical model. Next, at block211, a test is performed to ascertain whether or not the second plurality of sampled and recorded response times deviates from the set of predicted response times by at least a user defined amount. When the second plurality of sampled and recorded response times deviates from the set of predicted response times by at least a user defined amount, then the I/O device is switched to an alternate I/O device (block213), or an alert is triggered (block215), or both. For example, an alert in the form of an email or electronic message could be triggered if the second plurality of sampled and recorded response times deviates from the set of predicted response times by N standard deviations, or if an observed I/O response time was at least M times longer than predicted by the mathematical model. Illustratively, M and N are positive integers greater than one. For example, N could be three and M could be four. The negative branch from block211leads back to block207.

The foregoing exemplary embodiments may be provided in the form of computer-implemented processes and apparatuses for practicing those processes. The exemplary embodiments can also be provided in the form of computer program code containing instructions embodied in tangible media, such as floppy diskettes, CD ROMs, hard drives, or any other computer-readable storage medium, wherein, when the computer program code is loaded into and executed by a computer (such as, for example, one or more processing nodes100.i ofFIG. 1), the computer becomes an apparatus for practicing the exemplary embodiments. The exemplary embodiments can also be provided in the form of computer program code, for example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein, when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing the exemplary embodiments. When implemented on a general-purpose microprocessor, the computer program code segments execute specific microprocessor machine instructions. The computer program code could be implemented using electronic logic circuits or a microchip.