Preserving state information of a storage subsystem in response to communication loss to the storage subsystem

A storage subsystem has a plurality of storage devices. An indication of failure of at least one of the plurality of storage devices is detected. In response to detecting the indication of failure, monitoring is performed for a further condition. According to the monitored further condition, it is determined whether the at least one storage device has failed or whether communication has been lost to the storage subsystem. In response to determining that communication has been lost, state information of the storage subsystem is preserved to enable restoration of the storage subsystem after communication to the storage subsystem is recovered.

BACKGROUND

A storage subsystem can be implemented with multiple (e.g. an array of) physical storage devices. A storage volume, which is a logical unit of data, can be implemented across plural physical storage devices. One reason for implementing a storage volume across plural physical storage devices is to provide redundancy, such as redundancy based on mirroring data (in which data of a first storage device is copied to a second storage device to provide redundancy). Another form of redundancy is parity-based redundancy (in which parity information of data stored in a group of storage devices is kept in another storage device). By providing redundancy, if a physical storage device (or any part of a physical storage device) from the multiple physical storage devices were to fail, then the data stored in the storage subsystem can still be recovered under certain conditions.

With a conventional storage subsystem, it may be difficult to recover from a failure condition in which communication is lost to the entire storage subsystem. Typically, when communication to an entire storage subsystem is lost, a manual process is used to attempt to recover the storage subsystem. Such a manual process of storage subsystem recovery is labor-intensive and time-consuming.

DETAILED DESCRIPTION

When access (communication) is lost to a storage subsystem that includes multiple storage devices, conventional systems typically mark all storage devices inside the storage subsystem as failed. To recover from such failure, a user typically has to run a management utility to determine which storage devices within the storage subsystem are functional and which storage devices are failed. Such a process can be time-consuming and inefficient.

In accordance with some embodiments, an automated recovery mechanism is provided to efficiently determine whether indications of failure of storage device(s) within a storage subsystem are caused by communication loss to the storage subsystem or is due to actual failure of one or more storage devices. If it is detected that communication loss has occurred to the storage subsystem, state information associated with the storage subsystem can be preserved such that when communication to the storage subsystem is subsequently recovered, the state of the storage subsystem can be restored. In this manner, a user would not have to manually run a utility to identify which storage devices are functional and which are failed, and to restore the system based on such manual identification.

As used here, the term “storage device” refers to a physical storage element, such as a disk-based storage element (e.g., hard disk drive, optical disk driver, etc.) or other type of storage element (e.g., semiconductor storage element). The multiple storage devices within a storage subsystem can be arranged as an array, in some exemplary implementations. More generally, a storage subsystem refers to a collection of storage devices, where such collection of storage devices can be contained within an enclosure (defined by an external housing of the storage subsystem). Alternatively, the storage devices of a storage subsystem can be located in multiple enclosures.

Storage volumes can be defined across multiple storage devices of a storage subsystem. A “storage volume” (or more simply “volume”) refers to a logical unit of data that is defined in the storage subsystem. A storage volume that is defined across multiple storage devices allows for redundancy to be provided. Redundancy can be based on mirroring of data, where data in a source storage device is copied to a mirror storage device (which contains a mirror copy of the data in the source storage device). In this arrangement, if an error causes data of the source storage device to be unavailable, then the mirror storage device can be accessed to retrieve the data.

Another form of redundancy is parity-based redundancy where actual data is stored across a group of storage devices, and parity information associated with the data is stored in another storage device. If data within any of the group of storage devices were to become inaccessible (due to data error or storage device fault or failure), the parity information can be accessed to reconstruct the data. Examples of parity-based redundancy schemes include RAID (redundant array of inexpensive disks) schemes, including the RAID-5 and RAID-6 schemes. An example of a mirroring redundancy scheme is the RAID-1 scheme. Although reference is made to RAID in this description, it is noted that some embodiments of the invention can be applied to other types of redundancy schemes, or to any arrangement in which a storage volume is implemented across multiple storage devices (whether redundancy is used or not).

FIG. 1illustrates an exemplary arrangement that includes a storage arrangement100that has a storage subsystem102connected by a link103to a storage controller104. The link103can be one of several different types of links. As examples, the link103can be a SCSI (Small Computer System Interface) link, a serial attached SCSI (SAS) link, a Fibre Channel link, and so forth.

The storage controller104is connected to a network106(e.g., local area network, storage area network, or other type of network) to allow client computers108to access the storage controller104. The client computers108can issue read, write, or other input/output (I/O) requests over the network106to the storage controller104. In response to such requests, the storage controller104accesses the storage subsystem102to perform the requested accesses. The client computers108can be user computers, or alternatively, the client computers108can be server computers that are accessible by user computers.

The storage subsystem102includes an array of storage devices110. A number of volumes can be defined across the storage devices110. In the example ofFIG. 1, two volumes are shown, including volume A and volume B. In an alternative implementation, just one volume can be defined across the storage devices110, or more than two volumes can be defined across the storage devices110. Although both volumes A and B are deployed across the same set of storage devices110, it is noted that in an alternative implementation, volume A can be implemented across a first collection of the storage devices110, and volume B can be implemented across a second collection of the storage devices110.

The storage subsystem102also includes an environmental monitoring unit (EMU)112, which gathers information relating to the storage subsystem102for reporting to the storage controller104over the link103. Examples of information that can be reported by the EMU112to the storage controller104include one or more of the following: temperature within the storage subsystem102, hot plug events (in which electronic components such as storage devices have been either removed or inserted into the storage subsystem102while the storage subsystem remains live), information relating to fans of a storage subsystem102, and so forth.

The storage devices110and EMU112(as well as other components of the storage subsystem102) can be contained in an enclosure defined by a housing (represented by the rectangular box surrounding the storage devices110and EMU112).

The storage controller104includes a monitoring module114according to some embodiments for monitoring for failures associated with the storage subsystem102. The failures associated with the storage subsystem102can be loss of communication to the storage subsystem102, or alternatively, can be failure of individual ones of the storage devices110.

The monitoring module114includes a timer116(which is described further below). The storage controller104also includes a memory118in which state information120associated with the storage subsystem102can be stored. The storage controller104also includes an interface124to allow communication with the storage subsystem102over the link103. The interface can include a network interface controller and any communication protocol stacks.

The storage controller104can be a processor-based system that includes a processor122. The monitoring module114can be a software module executable on the processor122. In a different embodiment, the storage controller104can be implemented with a hardware device such as an application specific integrated circuit (ASIC), programmable gate array (PGA), and so forth. In this case, the monitoring module114can be firmware embedded in the storage controller104, or alternatively the monitoring module114can be implemented as fixed circuitry in the storage controller104.

FIG. 2illustrates a general process according to an embodiment. The process ofFIG. 2can be performed by the monitoring module114in the storage controller104ofFIG. 1. The storage controller104detects (at202) an indication of failure of a storage device in the storage subsystem102. The indication of failure of a storage device can be in response to detecting loss of communication from a storage device. For example, the storage controller104may have submitted a request to a particular one of the storage devices110in the storage subsystem102. If the storage device fails to respond after some amount of time, then that is an indication of failure of the storage device.

A storage device can fail for various reasons. A physical component of the storage device may fail, such as failure of a power supply, failure of a mechanical part, and so forth. However, the indication of failure of a storage device may also be caused by loss of communication to the entire storage subsystem102. The loss of communication may be due to a loose cable connection, or due to a temporary transient condition associated with the link103. Loss of communication between the storage controller104and storage subsystem102can also be caused by failure of a communications interface in either the storage controller104or storage subsystem102.

When loss of communication occurs, it may be the case that none of the storage devices110are failed; rather, the storage devices110of the storage subsystem102may be fully functional, and the loss of communication may be a loss that can be recovered at a later point in time by repairing the link103or waiting for a temporary transient condition to dissipate. When loss of communication to the storage subsystem102occurs, it is undesirable to mark all storage devices110in the storage subsystem as failed, since doing so would preclude easy restoration of the storage subsystem102when communication to the storage subsystem102is later recovered.

In response to detecting an indication of failure of a storage device, the storage controller monitors (at204) for a further condition. In some embodiments, this further condition relates to whether or not the storage controller104is able to receive information from other parts of the storage subsystem102. If the storage controller104is unable to receive further information from other parts of the storage subsystem102, then that is an indication that loss of communication has occurred to the entire storage subsystem102, and therefore, no failure of any individual storage device should be made in response to the detected loss of communication.

Based on the monitored further information, the storage controller104determines (at206) whether the storage device has failed or communication has been lost to the storage subsystem. If the communication to the storage subsystem is lost, then the state information of the storage subsystem is preserved (at208) for later use when communication to the storage subsystem is recovered. This state information can be stored as state information120in the memory118of the storage controller104(FIG. 1).

FIG. 3is a process of a more specific embodiment of the invention. The storage controller detects (at302) an indication of failure of a storage device. In response to such indication, the storage controller saves (at304) state information of one or more storage volumes defined in the storage subsystem102. The state information of a storage volume includes one or more of the following: information regarding which storage device(s) of the storage volume was previously marked as failed; information relating to which storage device(s) of the storage volume has been replaced; the content of a posted write buffer for the storage volume; information relating to the status of redundancy for the storage volume (e.g., status of building of RAID-5 or RAID-6 parity information, status of mirroring of data for RAID-1 redundancy); and so forth.

A posted write buffer refers to a buffer in the storage controller104that is maintained for write requests to a storage volume. When a write request is received from a client device110for the storage volume, the write request is stored in the posted write buffer. The posted write buffer is typically a memory that is battery-backed to allow the content of the memory to be preserved even if power of the storage controller104is lost. When a write request is posted to the posted write buffer, a response can be returned to the requesting client computer that the write has been completed, even though the write may not actually have been performed to the storage subsystem102. The write requests in the posted write buffer can be gradually flushed to the storage subsystem102.

In response to the indication of failure of the storage device (detected at302), the timer116(FIG. 1) of the monitoring module114is started (at306). Next, the storage controller104monitors for a further condition. In this case, the further condition includes whether or not information from the EMU112(FIG. 1) of the storage subsystem102is subsequently received. Thus, the storage controller determines (at308) whether information from the EMU112has been received. If not, the storage controller104next determines (at310) whether the timer has expired (the timer has counted a transpired time, a transpired number of clock cycles, etc.). If not, the storage controller104continues to wait for information from the EMU112. If information from the EMU112is received, then it is confirmed that there is no communication loss to the storage subsystem102, but rather, that an individual storage device has actually failed. In this case, the storage controller104marks (at312) the individual storage device as failed.

However, if the timer expires (310) before any information is received from the EMU112, then that is confirmation that communication loss has likely occurred between the storage controller104and the storage subsystem102. In this case, the storage controller104indicates (at314) that communication has been lost to the storage subsystem102. The state information of the storage volume(s) that has been saved (at304) is preserved.

The storage controller104next determines (at316) if communication to the storage subsystem has been recovered. If so, the storage volume(s) in the storage subsystem102is restored (at318) using the saved state information.

In this way, an automated and efficient mechanism is provided to distinguish between loss of communication to a storage subsystem versus failure of individual storage devices. If loss of communication to an entire storage subsystem is detected, then individual storage devices are not marked as failed in response to such detected loss of communication. By preserving state information associated with the storage subsystem in response to loss of communication to the storage subsystem, the storage volume(s) of the storage subsystem can later be recovered when communication to the storage subsystem is recovered.

Instructions of software described above (including the monitoring module114ofFIG. 1) are loaded for execution on a processor (such as processor122inFIG. 1). The processor includes microprocessors, microcontrollers, processor modules or subsystems (including one or more microprocessors or microcontrollers), or other control or computing devices. As used here, a “processor” can refer to a single component or to plural components (e.g., one CPU or multiple CPUs).