In-line announcement of impending critical events within a distributed storage environment

A method for announcing impending critical events within a distributed storage environment is disclosed. In one embodiment, such a method analyzes, at a storage system, status of various storage system components to predict an impending critical event to occur at the storage system. Predicting the critical event may include calculating an amount of time before occurrence of the impending critical event. The method then communicates, from the storage system to a host system, over an in-band communication channel used to carry I/O traffic between the host system and the storage system, one or more of the impending critical event and the amount of time. This will ideally enable the host system to take mitigating actions before the critical event occurs. A corresponding system and computer program product are also disclosed.

BACKGROUND

Field of the Invention

This invention relates to systems and methods for announcing impending critical events for collaborative data hardening within a distributed storage environment.

Background of the Invention

Traditionally, when critical events occur within an enterprise class storage array, they happen in a vacuum without host systems being informed of what is happening at the storage array. This means that host systems are typically completely unaware of impending issues in a storage array. Currently, the two most common critical events that adversely impact host systems are input power loss and critical over temperature of storage arrays. Historically, it was believed that communication with hosts regarding these issues was neither practical nor possible. In the case of losing input power at the storage array, it was assumed that connected hosts would lose input power as well and thus did not need to take any action. In modern data center environments, this is no longer necessarily the case since it is quite possible for a host system and a storage system to be on completely separate power distribution systems.

In the event of critical over temperature on a storage array, it was typically assumed that data center monitoring and management protocols would already be taking actions to address the over temperature and shut down host systems. If the storage array needed to shut down, the host systems would most likely already be shut down. Experience with modern data centers, however, in which many are unattended “lights out” facilities, has proven that this is not the case and many times a storage array will detect an over temperature condition well before a customer has any idea that the data center is even having cooling problems.

While many third party/external solutions exist for monitoring data center environments, the tight integration between storage arrays and host systems requires a new approach to how this information is communicated. Ideally, this approach will not rely on external monitoring which may or may not detect issues or critical events occurring on storage arrays in time to take mitigating actions. Such an approach is disclosed herein.

SUMMARY

The invention has been developed in response to the present state of the art and, in particular, in response to the problems and needs in the art that have not yet been fully solved by currently available systems and methods. Accordingly, systems and methods are disclosed to announce impending critical events within a distributed storage environment. The features and advantages of the invention will become more fully apparent from the following description and appended claims, or may be learned by practice of the invention as set forth hereinafter.

Consistent with the foregoing, a method for announcing impending critical events within a distributed storage environment is disclosed. In one embodiment, such a method analyzes, at a storage system, status of various storage system components to predict an impending critical event to occur at the storage system. Predicting the critical event may include calculating an amount of time before occurrence of the impending critical event. The method then communicates, from the storage system to a host system, over an in-band communication channel used to carry I/O traffic between the host system and the storage system, one or more of the impending critical event and the amount of time. This will ideally enable the host system to take mitigating actions before the critical event occurs.

A corresponding system and computer program product are also disclosed and claimed herein.

DETAILED DESCRIPTION

The present invention may be embodied as a system, method, and/or computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

The computer readable program instructions may execute entirely on a user's computer, partly on a user's computer, as a stand-alone software package, partly on a user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter scenario, a remote computer may be connected to a user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

Referring toFIG. 1, one example of a network environment100is illustrated. The network environment100is presented to show one example of an environment in which systems and methods in accordance with the invention may be implemented. The network environment100is presented by way of example and not limitation. Indeed, the systems and methods disclosed herein may be applicable to a wide variety of network environments, in addition to the network environment100shown.

As shown, the network environment100includes one or more computers102,106interconnected by a network104. The network104may include, for example, a local-area-network (LAN)104, a wide-area-network (WAN)104, the Internet104, an intranet104, or the like. In certain embodiments, the computers102,106may include both client computers102and server computers106(also referred to herein as “host systems”106). In general, the client computers102initiate communication sessions, whereas the server computers106wait for requests from the client computers102. In certain embodiments, the computers102and/or servers106may connect to one or more internal or external direct-attached storage systems112(e.g., arrays of hard-disk drives, solid-state drives, tape drives, etc.). These computers102,106and direct-attached storage systems112may communicate using protocols such as ATA, SATA, SCSI, SAS, Fibre Channel, or the like.

The network environment100may, in certain embodiments, include a storage network108behind the servers106, such as a storage-area-network (SAN)108or a LAN108(e.g., when using network-attached storage). This network108may connect the servers106to one or more storage systems110, such as arrays110aof hard-disk drives or solid-state drives, tape libraries110b, individual hard-disk drives110cor solid-state drives110c, tape drives110d, CD-ROM libraries, or the like. To access a storage system110, a host system106may communicate over physical connections from one or more ports on the host106to one or more ports on the storage system110. A connection may be through a switch, fabric, direct connection, or the like. In certain embodiments, the servers106and storage systems110communicate using a networking standard such as Fibre Channel (FC).

Referring toFIG. 2, one embodiment of a storage system110acontaining an array of hard-disk drives204and/or solid-state drives204is illustrated. The internal components of the storage system110aare shown since certain functionality in accordance with the invention may be implemented within such a storage system110a. As shown, the storage system110aincludes a storage controller200, one or more switches202, and one or more storage drives204, such as hard disk drives204and/or solid-state drives204(such as flash-memory-based drives204). The storage controller200may enable one or more hosts106(e.g., open system and/or mainframe servers106) to access data in the one or more storage drives204.

In selected embodiments, the storage controller200includes one or more servers206. The storage controller200may also include host adapters208and device adapters210to connect the storage controller200to host devices106and storage drives204, respectively. Multiple servers206a,206bmay provide redundancy to ensure that data is always available to connected hosts106. Thus, when one server206afails, the other server206bmay pick up the I/O load of the failed server206ato ensure that I/O is able to continue between the hosts106and the storage drives204. This process may be referred to as a “failover.”

In selected embodiments, each server206may include one or more processors212and memory214. The memory214may include volatile memory (e.g., RAM) as well as non-volatile memory (e.g., ROM, EPROM, EEPROM, hard disks, flash memory, etc.). The volatile and non-volatile memory may, in certain embodiments, store software modules that run on the processor(s)212and are used to access data in the storage drives204. The servers206may host at least one instance of these software modules. These software modules may manage all read and write requests to logical volumes in the storage drives204.

One example of a storage system110ahaving an architecture similar to that illustrated inFIG. 2is the IBM DS8000™ enterprise storage system. The DS8000™ is a high-performance, high-capacity storage controller providing disk storage that is designed to support continuous operations. Nevertheless, the systems and methods disclosed herein are not limited to the IBM DS8000™ enterprise storage system110a, but may be implemented in any comparable or analogous storage system110, regardless of the manufacturer, product name, or components or component names associated with the system110. Furthermore, any storage system that could benefit from one or more embodiments of the invention is deemed to fall within the scope of the invention. Thus, the IBM DS8000™ is presented only by way of example and is not intended to be limiting.

Referring toFIG. 3, as previously mentioned, traditionally, when critical events occur within a storage system110such as that illustrated inFIG. 2, they happen in a vacuum without host systems106being informed of what is happening on the storage system110. This means that host systems106are typically completely unaware of impending issues on the storage system110. Currently, the two most common critical events that adversely impact host systems106are input power loss and critical over temperature of connected storage systems110. Historically, it was believed that communication with host systems106regarding these issues was neither practical nor possible. In the case of losing input power at the storage system110, it was assumed that connected host systems106would lose input power as well and thus no action was needed. In modern data center environments, however, this is no longer necessarily the case. In such environments, it is quite possible for a host system106and storage system110to be on completely separate power distribution systems.

In the event of critical over temperature on a storage system110, it was typically assumed that data center monitoring and management protocols would already be taking actions to address the over temperature and shut down host systems106. If a storage system110needed to shut down, the host systems106would most likely already be shut down. Experience with modern data centers, however, in which many are unattended “lights out” facilities, has proven that this is not the case. In many cases, a storage system110will detect an over temperature condition well before a user or data center monitoring tool has any idea that the data center is having cooling problems.

While many third party/external solutions exist for monitoring data center environments, the tight integration between storage systems110and host systems106requires a new approach to how this information is communicated. Ideally, this approach will not rely on external monitoring which may or may not detect issues or critical events occurring on storage systems110. One such approach is illustrated inFIG. 3and disclosed hereinafter.

As shown inFIG. 3, a host system106and storage system110may be equipped or programmed with various modules to provide various features and functions in accordance with the invention. These modules may be implemented in hardware, software, firmware, or combinations thereof. These modules may include one or more of an analysis module300, mitigation module302, status monitoring module304, prediction module306, and communication module308. The modules are presented by way of example and are not intended to represent an exhaustive list of modules that may be used to implement functionality in accordance with the invention. Systems and methods in accordance with the invention may include more or fewer modules than those illustrated, or the functionality of the modules may be organized differently.

As shown, a status monitoring module304on the storage system110may monitor status of various storage system components. Such status may include power status, temperature status, operational status, or the like of various storage system components, which may include any or all components (internal or external) relied upon by the storage system110for operation.

A prediction module306in or in association with the storage system110may use the status information gathered by the status monitoring module304to predict critical events that may occur on or in association with the storage system110. For example, if temperature of a storage system component is trending upward, the prediction module306may predict that a critical over temperature is imminent or on the horizon. Similarly, if input power to the storage system110is lost, the prediction module306may predict that the storage system110will need to shut down (e.g., in the event the storage system110is currently running on backup power). In yet other cases, the prediction module306may predict that a preferred communication path between the host system106and the storage system110will go down and become unavailable. Using the status information, including changes in the status information over time, the prediction module306may estimate an amount of time it will take for the critical event to occur and potentially make the storage system110unavailable.

Once a critical event and/or amount of time associated with the critical event are determined, a communication module308may communicate the critical event and/or amount of time to a host system106in operable communication with the storage system110. In certain embodiments, the communication module308may utilize an in-band communication channel used to carry I/O traffic between the host system106and the storage system110. Furthermore, the communication module308may utilize or piggyback on an existing standard for transferring data between the host system106and the storage system110.

For example, the communication module308may in certain embodiments leverage existing optional or vendor-specific data fields or commands within the SCSI specification to actively inform host systems106about impending critical events occurring on or in association with the storage system110. For example, the communication module308may utilize the sense field descriptor or a vendor-specific command section within a SCSI header to communicate information. By using the existing SCSI specification, each and every time that a data packet is transmitted between the storage system110and the host system106, the host system106may potentially be informed of an impending critical event on the storage system110, and/or an estimated amount of time before the critical event will occur. Leveraging the SCSI specification enables systems and methods in accordance with the invention to be used on legacy host systems106without committing major development resources since the added functionality may be implemented as part of a software component on the legacy system without requiring hardware alterations.

As shown inFIG. 3, SCSI commands310, such as read, write, or Test Unit Ready commands310, may be regularly transmitted between a host system106and storage system110to read or write data, or to determine if a device106,110is ready to receive or transmit data. As further shown inFIG. 3, when a SCSI command310is received by the storage system110, the storage system110may return a SCSI acknowledgment312to the host system106. This SCSI acknowledgment312may inform the host system106that the storage system110received or successfully processed the SCSI command310. Systems and methods in accordance with the invention may, in certain embodiments, embed or provide information within optional data fields (in the header, for example) of the SCSI acknowledgment312to inform the host system106that a critical event is impending on the storage system110. In certain embodiments, an estimated amount of time before the critical event occurs may also be provided in the SCSI acknowledgment312.

Alternatively, the SCSI acknowledgment312may simply alert the host system106that something on the storage system110requires attention (using, for example, an “unit attention” or “check condition” indicator) and the host system106may send additional SCSI commands to the storage system110to request more details. The storage system110may then inform the host system106of the impending critical event, the estimated amount of time before occurrence of the critical event, and/or other details associated with the critical event.

Upon receiving a SCSI acknowledgment312, an analysis module300in the host system106may analyze the SCSI acknowledgment312and any information associated with critical events. This may include analyzing an amount of time until occurrence of the critical event, storage system components that are associated with the critical event, the nature of the critical event, potential problems or data loss that may result from the critical event, and or the like. The mitigation module302may then determine mitigating actions to take to mitigate the critical event. Such mitigating actions may include, for example, shutting down the host system106, hardening data on the host system106and/or storage system110, unmounting volumes on the storage system110, disabling write caching in the host system106(since data in the write cache may have no where to go if the storage system110is unavailable), flushing buffers on the host system106, communicating with the storage system110over an alternative path, and/or the like. In certain embodiments, the mitigation module302may determine mitigating actions that may be completed before the storage system110becomes unavailable. Once appropriate mitigating actions are determined, the mitigation module302may take these actions.

In certain embodiments, in cases where the mitigation module302may not have time to implement all desired mitigating actions before the critical event occurs, the mitigation module302may request additional time from the storage system110. For example, if a critical event is estimated to occur in thirty seconds, but mitigating actions cannot be completed within that time frame, the mitigation module302may transmit a request (e.g., a vendor-specific SCSI command) to the storage system110to request that actions (e.g. shut down, etc.) associated with the critical event be delayed beyond the time that they would normally occur. If possible, the storage system110will delay these actions to provide time for the host system106to perform mitigating actions.

Referring toFIG. 4, a first scenario of actions and interactions by a host system106and storage system110is illustrated. As shown, the storage system110initially monitors400the status of various storage system components. The storage system110analyzes402the status information to predict a critical event that is impending on the storage system110. The storage system110also calculates404an amount of time before occurrence of the critical event.

Meanwhile, the host system106transmits406a SCSI command310to the storage system110during the course of normal operations. This SCSI command310may be a read, write, or Test Unit Ready command, for example. Upon receiving408the SCSI command310, the storage system110processes408the command. The storage system110then returns410a SCSI acknowledgment312that acknowledges that the SCSI command310was processed. The storage system110incorporates410, into this SCSI acknowledgment312, information about a critical event impending on the storage system110and/or an amount of time before occurrence of the critical event. In certain embodiments, this may be accomplished by embedding the information into optional or vendor-specific data fields of the SCSI acknowledgment312.

The host system106receives412the SCSI acknowledgment312and analyzes412the information in the acknowledgment312. This will inform the host system106of the impending critical event and possibly the amount of time before it occurs. At this point, the host system106may take 414 mitigating actions to minimize or reduce affects of the critical event. This may include, for example, shutting down the host system106, hardening data on the host system106and/or storage system110, unmounting volumes on the storage system110, disabling write caching in the host system106, flushing buffers on the host system106, communicating with the storage system110over an alternative path, and/or the like.

Referring toFIG. 5, a second scenario of actions and interactions by a host system106and storage system110is illustrated. This scenario is similar to that illustrated inFIG. 4(many of the steps are the same), except that the host system106, upon receiving and analyzing412the SCSI acknowledgment312, determines that additional time may be needed to take desired mitigating actions. Upon making this determination, the host system106sends500a request to the storage system110to postpone a shutdown (of the storage system110as a whole or specific storage drives204or arrays of storage drives204on the storage system110) so that mitigating actions may be completed. The storage system110receives502the request and executes502the request if possible. The host system106takes the desired mitigating actions on the host system106. The scenario presented inFIG. 5(as well as the other Figures) enables both the host system106and the storage system110to be cognizant of situations or conditions on each other in substantially real time. This, in turn, enables the host system106and storage system110to make intelligent choices as to the best way to mitigate a critical event as quickly as possible.

Referring toFIG. 6, a third scenario of actions and interactions by a host system106and storage system110is illustrated. This scenario is also similar to that illustrated inFIG. 4, except that the storage system110returns600a SCSI acknowledgment312that incorporates a “check condition” or “unit attention” indicator. Upon receiving the SCSI acknowledgment312, the host system106analyzes412the SCSI acknowledgment312and detects the “check condition” or “unit attention” indicator. In response, the host system106sends602a request to the storage system110for additional information. The storage system110receives604this request and returns604the desired information to the host system106. For example, the storage system110may inform the host system106of an impending critical event and/or an amount of time before occurrence of the critical event. Using this information, the host system106may take 414 mitigating actions to minimize or reduce any impact of the critical event.

Referring toFIG. 7, a fourth scenario of actions and interactions by a host system106and storage system110is illustrated. This scenario is also similar to that illustrated inFIG. 4, except for actions performed by the host system106after receiving and analyzing412the SCSI acknowledgment312. As shown, after receiving and analyzing412a SCSI acknowledgment312that indicates that a critical event is impending on the storage system110, the host system106waits700for a selected amount of time prior to initiating mitigating actions. The purpose of this wait period is to see if the critical event is resolved on the storage system110on its own, which may be indicated in SCSI acknowledgments312received by the host system106from the storage system110. In the illustrated scenario, the critical event is resolved702on the storage system110and the host system106performs no further actions. On the other hand, if the critical event is not resolved during the wait period, the host system106may initiate mitigating actions and/or request delay of actions on the storage system110.