Storage system, control apparatus and control method therefor

In a storage system, a first reboot controller in a first control apparatus causes a second control apparatus to reboot, when it is detected that a second control apparatus has stopped access operations. The first reboot controller also places a boot event record in a non-volatile storage device of the second control apparatus to indicate that the rebooting of the second control apparatus has been caused by the first control apparatus. After that, a second reboot controller in the second control apparatus causes at least the first control apparatus to reboot while keeping intact the cache data stored in a cache memory of the first control apparatus, when the access controller of the first control apparatus is stopped while the second control apparatus is rebooted, and when a boot event record is found in the non-volatile storage device of the second control apparatus.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2010-192070, filed on Aug. 30, 2010, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein relate to a storage system, as well as to a control apparatus and a control method therefor.

BACKGROUND

Data storage systems formed from a plurality of large-capacity storage devices such as hard disk drives (HDDs) have widely been used in recent years. Typical data storage systems incorporate a number of storage devices, combined with a control device to control access to those storage devices. Some of such storage systems employ two or more control devices to provide redundancy-protected access paths to the storage devices for more reliable operation.

The control devices may have a local cache memory to cache the data stored in storage devices so as to increase the speed of access to the storage devices. For example, a storage system may employ two control devices with individual cache memories, and those control devices may be configured to duplex their content by storing each other's cache data in addition to their own. (See, for example, Japanese Laid-open Patent Publication No. 10-105467.)

The above-exemplified storage system is duplexed, or dual-redundant, in terms of both control device functionality and cache data content. Suppose, for example, one of the two control devices has stopped due to some problem and is thus initialized. Because cache data is duplexed, the initialized control device can reload its local cache memory with a copy of valid cache data from the other control device. This valid cache data in the cache memory permits the initialized control apparatus to continue access operation to the storage devices without slowing down the access speed.

For another example, some storage systems further have a supervisory unit in addition to the duplexed control device functionality and cache data content. This supervisory unit operates independently of the control devices to supervise their activities. When it is detected that access controllers have stopped in both control devices, the supervisory unit causes the two control devices to reboot in “restoration mode.” In this restoration mode, the cache memory in each control device is kept powered during the rebooting, not to lose the cache data stored therein. As a result of rebooting both control devices in restoration mode, their respective access controllers can resume access control operations on the storage devices by using the latest cache data. (See, for example, Japanese Laid-open Patent Publication No. 2004-206239.)

For yet another example of the multiple control device architecture, a proposed storage system performs exclusive control and space reservation of cache memories on an individual control device basis, by using configuration management data of each cache memory and information that indicates the association between cache memories and control devices. This method enables allocating storage spaces of a single cache memory to write cache data, its copy for redundancy, and read cache data in a desired way. (See, for example, Japanese Laid-open Patent Publication No. 2008-047029.)

The storage systems discussed above are protected by duplexing both control device functionality and cache data content. For this reason, even in the event that one of the two control devices is stopped and initialized due to some problem, the initialized control device can reload its local cache memory with valid cache data supplied from the other control device that is alive. However, it is still possible that the other control device may also encounter some anomalies and thus stop during the ongoing initialization process of the failed control device. If this happens, the initialized control apparatus cannot receive latest cache data from the other control apparatus that is in need of initialization. The first-initialized control apparatus may still be able to restart its access control operations without reloading cache entries, but this means that the control apparatus loses the benefit of having the latest cache data in its cache memory.

SUMMARY

According to an aspect of the invention, there is provided a storage system including a storage apparatus to store data and first and second control apparatuses to control access to the storage apparatus. Each of the first and second control apparatuses includes: a cache memory to cache the data in the storage apparatus, a non-volatile storage device, and an access controller to make access to the storage apparatus by using cache data in the cache memory in response to an access request from a host device, while managing the respective cache memories of the first and second control apparatuses such that each other's cache data is mirrored therein. Further, a first reboot controller is disposed in the first control apparatus, which causes, upon detection that the access controller in the second control apparatus is stopped, the second control apparatus to reboot and places a boot event record in the non-volatile storage device of the second control apparatus to indicate that the rebooting of the second control apparatus has been caused by the first control apparatus. A second reboot controller is disposed in the second control apparatus, which causes the first control apparatus to reboot while keeping intact the cache data stored in the cache memory of the first control apparatus, when the access controller of the first control apparatus is stopped while the second control apparatus is rebooted, and when the non-volatile storage device of the second control apparatus stores the boot event record indicating that the rebooting of the second control apparatus has been caused by the first control apparatus.

DESCRIPTION OF EMBODIMENTS

Several embodiments of the present invention will be described below with reference to the accompanying drawings, wherein like reference numerals refer to like elements throughout.

(a) First Embodiment

FIG. 1illustrates an example of a storage system according to a first embodiment. The illustrated storage system1is formed from a storage apparatus10and a plurality of control apparatuses. WhileFIG. 1depicts only two control apparatuses20aand20bfor exemplary purposes, the storage system1may actually have three or more control apparatuses.

The storage apparatus10may include a plurality of storage media such as HDDs to store data. First and second control apparatuses20aand20bcontrol access operations to this storage apparatus10. For example, the first and second control apparatuses20aand20bmake access to the storage apparatus10in response to access requests that host devices (not illustrated) issue to access the storage apparatus10.

The two control apparatuses20aand20bin the storage system1have similar or the same structures. Specifically, the first control apparatus20aincludes an access controller21a, a cache memory22a, reboot controllers23aand24a, and a non-volatile storage unit25a. Likewise, the second control apparatus20bincludes an access controller21b, a cache memory22b, reboot controllers23band24b, and a non-volatile storage unit25b. The two access controllers21aand21bprovide the same functionalities. Likewise, the reboot controllers23aand23bprovide the same functionalities, as do the reboot controllers24aand24b. Because of this similarity, the following description of processing functions will focus on the first control apparatus20aand avoid repeating similar explanations for the second control apparatus20b.

The access controller21amakes access to the storage apparatus10in response to an access request from a host device, while managing the cache memory22ain the first control apparatus20ato hold a partial copy of data stored in the storage apparatus10. For example, when a data read request to the storage apparatus10is received from a host device, the access controller21areads the requested data from the storage apparatus10and sends the read data back to the requesting host device. Here the requested data may happen to be stored in an entry of the cache memory22a. If this is the case, the access controller21areads that data from the cache memory entry, instead of executing a read access to the storage apparatus10. When, on the other hand, a data write request to the storage apparatus10is received from a host device, the access controller21awrites the specified write data to both the cache memory22aand storage apparatus10.

The access controller21aalso causes the second control apparatus20bto manage its cache memory22bsuch that the two cache memories22aand22bfurther store a copy of each other's cache data. In other words, the two cache memories22aand22bare mirrored in each other. For example, the access controller21apasses the data stored in its local cache memory22ato its peer access controller21bin the second control apparatus20b, so that the same data is entered to the cache memory22b. Similarly, the access controller21bin the second control apparatus20balso causes the first control apparatus20ato make its cache memory22aconsistent with the peer cache memory22b. This feature of mutual mirroring enables the two control apparatuses20aand20bto keep their cache data in a duplexed state.

The reboot controller23ahas a function of detecting that the access controller21bis stopped in the second control apparatus20bdue to some anomalies or other event. For example, the reboot controller23amay be designed to detect disruption of communication with the second control apparatus20b, which indicates that the access controller21bhas stopped. Upon detection of such stop state of the access controller21b, the reboot controller23amakes the second control apparatus20breboot. The reboot controller23athen places a boot event record26bin its non-volatile storage unit25bof the second control apparatus20bto indicate that the rebooting of the second control apparatus20bhas been caused by a command from outside the second control apparatus itself (e.g., from the first control apparatus).

The above boot event record26bmay be implemented in the form of, for example, a flag that stays resident in the non-volatile storage unit25b. In this case, the reboot controller23achanges the state of the flag when rebooting the second control apparatus20b, so that the flag indicates the fact that other control apparatus has commanded the second control apparatus20bto reboot.

In addition to the boot event record, the reboot controller23amay further record a piece of cache state information27ain its local non-volatile storage unit25awhen rebooting the second control apparatus20bin response to stop detection of the access controller21b. The purpose of this cache state information27ais to indicate that the cache memory22ain the first control apparatus20acontains the latest cache data.

The cache state information27amay be implemented in the form of, for example, a flag that stays resident in the non-volatile storage unit25b, similarly to the foregoing boot event record26a. In this case, the reboot controller23achanges the state of the flag when rebooting the second control apparatus20b, so that the cache state information27aindicates the fact that the cache memory22acontains the latest cache data.

Another reboot controller24ain the first control apparatus20ahas a function of determining, at the time of bootup of the first control apparatus20a, whether the access controller21bin the second control apparatus20bis stopped. Upon bootup of the first control apparatus20a, its reboot controller24acauses at least the second control apparatus20bto reboot while keeping intact the current data in its cache memory22b, if the access controller21bin the second control apparatus20bis stopped at that time, and if the boot event record26ain the non-volatile storage unit25aindicates that the first control apparatus20ahas been commanded to reboot by other control apparatus.

Suppose, for example, that the access controller21ain the first control apparatus20aand the access controller21bin the second control apparatus20bhave stopped in that order.FIGS. 2 and 3illustrate how the storage system1works when two control apparatuses have successively stopped their operation.

The upper half ofFIG. 2depicts State1of the storage system1, in which the access controllers21aand21bin both control apparatuses20aand20bare working properly in controlling access operations to the storage apparatus10. It is now assumed that one control apparatus20bencounters a problem and thus stops its access controller21b. This fault event is detected by the reboot controller23ain the first control apparatus20aas depicted as State2in the lower half ofFIG. 2.

When it is detected that the access controller21bhas stopped, the reboot controller23amakes the second control apparatus20breboot. The reboot controller23aalso places a boot event record26bin the non-volatile storage unit25bin the second control apparatus20b, thereby indicating that the rebooting of the second control apparatus20bhas been caused by a command from other control apparatus than the second control apparatus20bitself (e.g., from the first control apparatus20a). The reboot controller23amay further record a piece of cache state information27ain the non-volatile storage unit25aof the first control apparatus20ato indicate that the cache memory22acontains the latest cache data.

The second control apparatus20breboots according to the command from the reboot controller23a. The second control apparatus20bnow flushes its cache memory22b, assuming that the cache memory22ain the first control apparatus20amaintains the latest cache data. More specifically, the cache memory22ain the first control apparatus20ais supposed to contain the same data as what the access controller21bhas accumulated in its local cache memory22buntil the rebooting of the second control apparatus20boccurs. The validity of this “mirrored cache data” in the first control apparatus20ais ensured as long as the second control apparatus20bis operational at the time of rebooting the second control apparatus20b. Accordingly, the access controller21bin the second control apparatus20bcan restart access control operations after reloading its local cache memory22bwith the mirrored cache data read out of the cache memory22ain the first control apparatus20a.

It may happen, however, that the access controller21ain the first control apparatus20astops before the rebooted second control apparatus20bbecomes ready to start access control operations. If this is the case, the access controller21bin the second control apparatus20bcannot read the mirrored cache data from the cache memory22ain the first control apparatus20a.

To overcome the above-described situation, the reboot controller24bcauses at least the first control apparatus20ato reboot while maintaining the data in its cache memory22a, as depicted as State3in the upper half ofFIG. 3. The reboot controller24bperforms this control if the access controller21ain the first control apparatus20astops when the second control apparatus20bis in the rebooting process, and if the non-volatile storage unit25bin the second control apparatus20bhas a boot event record26b. Then as depicted as State4in the lower half ofFIG. 3, the first control apparatus20areboots in response to the command from the reboot controller24b, maintaining the data in its cache memory22aas originally stored before the rebooting. The access controller21bin the second control apparatus20bcan therefore restart access control operations by loading its local cache memory22bwith the mirrored cache data that is read out of the cache memory22aof the first control apparatus20a. Or alternatively, the access controller21ain the first control apparatus20amay solely restart access control operations to the storage apparatus10by using the mirrored cache data in its local cache memory22a.

When the access controller21bin the second control apparatus20bis stopped, the above-described processing prevents the first control apparatus20afrom losing the latest mirrored cache data in its cache memory22a, even if the access controller21astops before the second control apparatus20breboots and becomes ready to restart the stopped access controller21b.

The first control apparatus20areboots as seen in the lower half ofFIG. 3. In this State4, the access controller21a, and the access controller21bas well, can recognize that the latest cache data is retained in, for example, the cache memory22a, by testing whether the rebooted first control apparatus20ahas cache state information27ain its non-volatile storage unit25a.

As a variation of the above embodiment, the two control apparatuses20aand20bmay further be designed to reboot together, maintaining their respective cache memory data as is, in the case where their access controllers are both found stopped. This simultaneous reboot function is referred to herein as a “dual restoration function.” When this dual restoration function is implemented, State3in the upper half ofFIG. 3is modified such that the reboot controller24bin the second control apparatus20binitiates rebooting of the two control apparatuses20aand20bin the following way.

During the reboot process of the second control apparatus20b, the reboot controller24btests whether the access controller21ain the first control apparatus20ais stopped, and whether the local non-volatile storage unit25bcontains a boot event record26b. If the result is positive on both tests, the reboot controller24bcauses the second control apparatus20bto stop its local access controller21b. By stopping the access controller21bin the second control apparatus20b, the reboot controller24bproduces an environment where the first control apparatus20acan reboot with the current cache data in its cache memory22a.

When the access controller21bstops, the dual restoration function in the first control apparatus20aor second control apparatus20bdetects that both access controllers21aand21bhave stopped and causes the first and second control apparatuses20aand20bto reboot while keeping intact the data in their respective cache memories22aand22b. The dual restoration function thus prevents the first control apparatus20afrom losing the latest mirrored cache data in its cache memory22a.

The above-described first embodiment allows the second control apparatus20bto reboot without maintaining data in its cache memory22bin the case where the access controller21astill operates properly when the access controller21bstops. While it may be possible to maintain the data in the cache memory22bwhen rebooting the second control apparatus20b, some of the cache data could lose its up-to-dateness before the second control apparatus20breboots, depending on what the access controller21adoes during that time. For example, the access controller21amay succeed what the peer access controller21bhas been doing until the second control apparatus20bis commanded to reboot. In this case, the access controller21aperforms access control operations by itself, using the mirrored cache data stored in its cache memory22a, which renders, on the other hand, the cache data in the cache memory22bobsolete.

The next section will describe a storage system according to a second embodiment which enables one control apparatus to continue access control operations by itself when another control apparatus becomes inoperable.

(b) Second Embodiment

FIG. 4illustrates an example of an overall structure of a storage system according to a second embodiment. The illustrated storage system100has a plurality of HDDs as its constituent storage devices. The storage system100encloses those HDDs in drive enclosures (DEs)200. The storage system100also includes first and second controller modules (CMs)300aand300bto control access to HDDs constituting DEs200. Further the storage system100has two power supply units (PSUs)211and212to provide electric power to the DEs200, first CM300a, and second CM300b. The DEs200, as well as the PSUs211and212, may be located outside the storage system100. Storage devices for the storage system100are not limited to HDDs, but may be other devices such as solid state drives (SSDs).

The storage system100is connected to a host device120and a control terminal130. The host device120sends access requests to either the first CM300aor the second CM300bin the storage system100according to user inputs initiating access to HDDs in the DEs200. For example, Fibre Channel links are used to connect the host device120with the first CM300aand second CM300b.

The control terminal130is used by a system administrator of the storage system100to manage its operation. For example, the system administrator configures the first and second CMs300aand300bby using the control terminal130. The control terminal130is linked to those CMs300aand300bvia, for example, a local area network (LAN) cable.

The first and second CMs300aand300bindividually control access to HDDs in DEs200in response to access requests from the host device120. For example, the host device120may issue a request for reading data in an HDD. In response, the first CM300areads the requested data out of the HDD and sends it back to the host device120. The host device120may also issue a request for writing data to an HDD. In response, the first CM300awrites the specified data to the HDD. When requested, the second CM300balso reads and writes data in a similar way.

In addition to the above, the first and second CMs300aand300bhave a function of caching data in HDDs of the DEs200. To this end, the two CMs300aand300bare designed to exchange information and mirror each other's cache data. The first and second CMs300aand300balso monitor each other's activities and can reboot each other as necessary.

FIG. 5illustrates an example hardware configuration of CMs in the storage system. The illustrated first CM300ais formed from a RAID-on-Chip (RoC) device301a, a random access memory (RAM)302a, a solid state drive (SSD)303a, a LAN interface (LAN I/F)304a, channel adapters (CA)305aand306a, a Serial Attached SCSI expander (SAS EXP)307a, a field programmable gate array (FPGA)308a, a non-volatile RAM (NVRAM)309a, and a power supply circuit310a. Here, RAID means “Redundant Arrays of Inexpensive Disks,” and SCSI stands for “Small Computer System Interface.”

The second CM300balso has a similar hardware configuration. That is, the second CM300bis formed from an RoC device301b, a RAM302b, an SSD303b, a LAN interface304b, CAs305band306b, a SAS expander307b, an FPGA308b, a NVRAM309b, and a power supply circuit310b. These components are the counterparts of the RoC device301a, RAM302a, SSD303a, LAN interface304a, CA305a,306a, SAS expander307a, FPGA308a, NVRAM309a, and power supply circuit310ain the first CM300a. The following description of hardware configuration focuses on the first CM300a. The same description also applies to the second CM300b.

While not explicitly depicted, the RoC device301aincludes a central processing unit (CPU) to control the entire CM300a, and a SAS controller for the CPU to interface with SAS expanders307aand307b. The CPU in the RoC device301aalso has a function to communicate with its peer CPU in the RoC device301bof the second CM300b. The communication path between those two CPUs is designated by the symbol “P1.”

The RAM302aserves as temporary storage for at least part of the software programs that the CPU in the RoC device301aexecutes, as well as for various data that the CPU101needs to execute the programs. The RAM302aalso provides an area for caching data of HDDs in DEs200.

The SSD303aserves as secondary storage of the first CM300ato store programs that the RoC device301aexecutes, as well as for various data that it needs to execute the programs. As an alternative, HDDs or other non-volatile storage devices may also serve the same purpose.

The LAN interface304ais connected to the control terminal130via a LAN cable, allowing the RoC device301ato communicate data with the control terminal130.

The CAs305aand306aserve as an interface through which the host device120and the first CM300acan exchange data. Separate fiber optic cables are used to connect those CAs305aand306ato the host device120to provide redundant communication paths between the first CM300aand host device120. This configuration makes their communication more reliable.

The SAS expander307aplays intermediary roles in communication between the aforementioned SAS controller in the RoC device301aand SAS devices. To this end, the SAS expander307acontains a CPU (not illustrated) and a non-volatile memory (not illustrated) to store firmware programs for the CPU. The CPU in the SAS expander307aexecutes firmware programs to achieve, for example, “dual CM restoration,” i.e., rebooting RoC devices301aand301bin both CMs300aand300b, keeping intact their cache data in the RAM302a. It is noted that the CPU in the SAS expander307aoperates independently of the one in the RoC device301a. Even if the RoC device301astops due to some anomalies, this CPU can continue to operate as long as the SAS expander307ais powered.

In the example ofFIG. 5, the RoC device301ain the first CM300ahas a connection to DEs200, not only via its local SAS expander307a, but also via another SAS expander307bin the second CM300b. The RoC device301ais thus coupled to DEs200through two SAS expanders307aand307b. In other words, its access path to the DEs200is duplexed.

Similarly, the RoC device301bin the second CM300bhas a connection to DEs200, not only via its local SAS expander307b, but also via another SAS expander307ain the first CM300a. The RoC device301bis thus coupled to DEs200through two SAS expanders307aand307b. In other words, its access path to the DEs200is duplexed.

The FPGA308ain the first CM300aoffers the function of monitoring whether its local RoC device301ais operating properly. Similarly, the FPGA308bin the second CM300boffers the function of monitoring whether its local RoC device301bis operating properly. The two FPGAs308aand308bmay communicate to mutually supervise the health of two CMs.

When it is detected that RoC devices301aand301bhave stopped in both the first and second CMs300aand300b, the FPGA308aso notifies the SAS expander307a. In response to this notification from the FPGA308a, the CPU in the SAS expander307ainitiates “dual CM restoration” mentioned above.

The FPGA308aalso provides the function of rebooting its local RoC device301ain the first CM300awhen there is a request from the RoC device301aitself, SAS expander307a, or FPGA308bin the second CM300b. More specifically, the FPGA308areboots the RoC device301ain either “normal mode” or “restoration mode.” In normal mode, the FPGA308aclears data in the RAM302awhen rebooting the RoC device301a, by temporarily cutting power supply from the power supply circuit310ato the RoC device301aand RAM302a. In restoration mode, the FPGA308apermits the power supply circuit310ato keep supplying power to the RAM302aso as to maintain the current data in the RAM302a, while temporarily cutting power supply to the RoC device301awhen rebooting it. This restoration mode may be modified to maintain, not all data in the RAM302a, but only cache data when rebooting the RoC device301a.

As a variation of the above, the FPGA308amay request its peer FPGA308bin the second CM300bto reboot the RoC device301bin normal mode or restoration mode. When making this boot request, the FPGA308aspecifies which mode to use.

Further, the FPGA308ahas a function of writing data to and reading data out of the NVRAM309ain response to a request from the RoC device301a, SAS expander307a, or the FPGA308bin the second CM300b. It is noted that the FPGA308aoperates independently of the CPU in the RoC device301a, just as the CPU in the SAS expander307adoes. This means, for example, that the FPGA308acan continue its operation even if the RoC device301abecomes inoperable due to some anomalies.

The NVRAM309ais a non-volatile memory for storing various data objects used by the FPGA308aand RoC device301a. The power supply circuit310aprovides electric power to the components of the first CM300aunder the control of the FPGA308a.

FIG. 6illustrates an example hardware configuration of a control terminal. The foregoing control terminal130is implemented as a computer seen inFIG. 6. This computer includes a CPU131to control its entire system. The CPU131is connected to a RAM132and other various peripheral devices on a bus138.

The RAM132serves as primary storage of the computer. Specifically, the RAM132is used to temporarily store the whole or part of software programs that the CPU131executes, in addition to other various data objects that it manipulates at runtime.

Peripheral devices on the bus138include, for example, an HDD133, a graphics processor134, an input device interface135, an optical disc drive136, and a communication interface137. The HDD133serves as secondary storage of the computer. Specifically, the HDD133stores programs that the CPU131executes, together with various data files. Flash memory and other semiconductor memory devices may also be used as secondary storage, similarly to the HDD133.

The graphics processor134, coupled to a monitor134a, produces video images in accordance with drawing commands from the CPU131and displays them on a screen of the monitor134a. The monitor134amay be, for example, a cathode ray tube (CRT) display or a liquid crystal display.

The input device interface135is connected to input devices such as a keyboard135aand a mouse135band supplies signals from those devices to the CPU131. The mouse135bis a pointing device, which may be replaced with other kinds of pointing devices such as touchscreen, tablet, touchpad, and trackball.

The optical disc drive136reads out data encoded on an optical disc136a, by using a laser light. The optical disc136ais a portable data storage medium, the data recorded on which can be read as a reflection of light or the lack of same. The optical disc136amay be a digital versatile disc (DVD), DVD-RAM, compact disc read-only memory (CD-ROM), CD-Recordable (CD-R), or CD-Rewritable (CD-RW), for example.

The communication interface137is connected to the first and second CMs300aand300bvia LAN cables to communicate data with them.

The hardware configuration ofFIG. 6may also be applied to the host device120. In that case, however, the host device120has a communication interface that links to the first and second CMs300aand300bvia fiber optic cables.

FIG. 7is a block diagram illustrating an example of processing functions provided by CMs. The illustrated first CM300aincludes an In/Out (I/O) access controller321a, a restoration controller322a, a monitor323a, and a boot controller324a. Processing functions of the I/O access controller321aare implemented as a software program executed by a CPU in the RoC device301a. Processing functions of the restoration controller322aare implemented as a software program executed by a CPU in the SAS expander307a. Processing functions of the monitor323aand boot controller324aare implemented as part of the FPGA308a.

The second CM300bhas processing functions similar to those of the first CM300a. That is, the second CM300bincludes an I/O access controller321b, a restoration controller322b, a monitor323b, and a boot controller324b, respectively corresponding to the I/O access controller321a, restoration controller322a, monitor323a, and boot controller324aof the first CM300a. Because of this similarity, the following description of processing functions will focus on the first CM300a, instead of repeating explanations for two CMs300aand300b.

The processing functions of the second CM300bare implemented as software programs similarly to those of the first CM300a. Specifically, the I/O access controller321bof the second CM300bis implemented as a software program executed by a CPU in the RoC device301b, and the restoration controller322bis implemented as a software program executed by a CPU in the SAS expander307b. Processing functions of the monitor323band boot controller324bare implemented as part of the FPGA308b.

The I/O access controller321amanages the storage space of HDDs in DEs200, besides controlling access to those HDDs. Specifically, the host device120sends access requests to HDDs in the DEs200. The I/O access controller321areceives such a request via a CA305aor306aand executes the requested access via the SAS expander307aor307b.

The I/O access controller321acontrols access to data stored in HDDs in the DEs200, while caching the data in its local RAM302a. The I/O access controller321afurther mirrors the cached data from its local RAM302ato the peer RAM302bin the second CM300b.

The I/O access controller321aalso has a function of detecting, through the aforementioned communication path P1to the second CM300b, a stop state of the RoC device301bresulting from some anomalies. Upon detection, the I/O access controller321arequests the second CM300bto reboot its RoC device301bin restoration mode. The I/O access controller321amakes this request via its local boot controller324a. The RoC device301bin the second CM300bthus reboots in restoration mode. If there is no change in the cache data of the first CM300abefore the RoC device301bbecomes operational, the RoC device301bis allowed to resume its I/O access control without the need for loading the latest cache data from the first CM300a.

The second CM300bstores a boot flag332bin its NVRAM309b. This boot flag332b, when set to one, indicates that the CM has rebooted upon request from other CM. Along with the request to the second CM300bto reboot its RoC device301bin restoration mode, the I/O access controller321aalso requests the second CM300bvia the boot controller324ato change the boot flag332bin the NVRAM309bfrom zero to one. The boot flag332bis referenced by the I/O access controller321bwhen it begins access control operations of HDDs after the RoC device301bis rebooted in the second CM300b.

Also along with the request to the second CM300bto reboot its RoC device301bin restoration mode, the I/O access controller321aincrements a configuration data update count331astored in the NVRAM309ain the first CM300a. As will be discussed later, the I/O access controller321aconsults control data in a configuration data table during its control operation, and the configuration data update count331ais supposed to be incremented each time the configuration data table is changed. The second CM300balso has its own configuration data update count331bin its NVRAM309b. When the I/O access controller321aincrements the configuration data update count331a, the configuration data update count331bin the second CM300bis incremented synchronously. The two configuration data update counts331aand331bthus indicate the same value in normal situations.

The exception is when the second CM300bis requested to reboot its RoC device301ain restoration mode. While the first CM300aincrements its configuration data update count331a, the second CM300bis unable to do so for its own configuration data update count331bsince the second CM300bis right in the process of rebooting. Accordingly the two configuration data update counts331aand331bexhibit different values when the foregoing dual CM restoration process is executed to reboot RoC devices301aand301bin both the first and second CMs300aand300b. In this situation, the I/O access controllers321aand321bdetermine which of the two configuration data update counts331aand331bhas a larger value. Since the former configuration data update count331ais larger than the latter configuration data update count331b, the I/O access controllers321aand321brecognize that a newer cache data resides in the first CM300acorresponding to the former configuration data update count331a.

While not mentioned above, it is through the FPGA308athat the I/O access controller321awrites and reads the configuration data update count331band boot flag332ain the NVRAM309a.

When the monitor323afinds that RoC devices301aand301bin both the first and second CMs300aand300bhave stopped because of some anomalies or other reasons, the restoration controller322aexecutes a dual CM restoration process to reboot the two RoC devices301aand301bin restoration mode. Specifically, the restoration controller322arequests its local boot controller324ain the first CM300ato reboot the local RoC device301ain restoration mode. The restoration controller322aalso requests the boot controller324ato interact with its peer boot controller324bin the second CM300bso as to make the RoC device301breboot in restoration mode.

The monitor323akeeps track of whether the RoC device301ais working properly in the first CM300a. The monitor323aalso keeps track of whether the RoC device301bis working properly in the second CM300b, through its peer monitor323bin the second CM300b. The monitor323amay find that the RoC devices301aand301bhave both stopped working in the first and second CMs300aand300bdue to some anomalies. If this happens, the monitor323aso notifies the restoration controller322a.

The boot controller324amakes the RoC device301areboot in normal mode or restoration mode, in response to a request from the I/O access controller321ain the first CM300a, or the restoration controller322ain the first CM300a, or the boot controller324bin the second CM300b. In normal mode, the boot controller324aclears data in the RAM302awhen rebooting the RoC device301a, by temporarily cutting power supply from the power supply circuit310ato the RoC device301aand RAM302a. In restoration mode, on the other hand, the boot controller324apermits the power supply circuit310ato keep supplying power to the RAM302aso as to maintain the current data stored in the RAM302a, while temporarily cutting power supply to the RoC device301awhen rebooting it. This restoration mode may be modified to maintain, not all data in the RAM302a, but only cache data when rebooting the RoC device301a.

The boot controller324amay also be configured to request the boot controller324bin the second CM300bto make the RoC device301breboot in normal mode or restoration mode, in response to a request from the I/O access controller321aor restoration controller322a. Also, the boot controller324ahas a function of requesting its peer boot controller324bin the second CM300bto change the boot flag332bin NVRAM309b, when so requested by the I/O access controller321a. In addition, the boot controller324ahas a function of changing the boot flag332ain NVRAM309a, when so requested by the peer boot controller324bin the second CM300b.

The following section will now describe in greater detail the processing operation of the first and second CMs300aand300b. The first thing to discuss is how the I/O access controllers321aand321bcontrol access operations to HDDs in DEs200. The access control operation may be performed in either “redundant operation mode” or “solitary operation mode.” In redundant operation mode, both I/O access controllers321aand321bexecute access control tasks. In solitary operation mode, only one of the two I/O access controllers321aand321bundertakes the tasks.

FIG. 8illustrates access control in redundant operation mode. When both I/O access controllers321aand321boperate properly, each of them undertakes different volumes (storage spaces) in HDDs of a DE200. In the example ofFIG. 8, the I/O access controller321ain the first CM300acontrols access to volume Vol#0, while the I/O access controller321bin the second CM300bcontrols access to volume Vol#1. Here, the volumes Vol#0and Vol#1are different portions of the HDD storage spaces in the DE200.

In redundant operation mode, the I/O access controller321auses a local cache area341ain the RAM302ato cache data stored in volume Vol#0. Likewise, the I/O access controller321buses a local cache area341bin the RAM302bto cache data stored in volume Vol#1. For example, the I/O access controller321amay receive a request from the host device120to read data in volume Vol#0. If the local cache area341acontains a cache entry corresponding to the requested data, the I/O access controller321areads data of that entry from the local cache area341aand sends it to the host device120. The I/O access controller321amay also receive a request from the host device120to write specific data. If the local cache area341ahas an existing entry for the write address, the I/O access controller321afirst updates that cache entry with write data specified by the host device120. Afterwards the I/O access controller321aupdates data in a relevant HDD in the DE200with the same write data.

Further, the data in the local cache area of one CM is duplexed in RAM of the other CM. In the example illustrated inFIG. 8, the second CM300bhas a mirror cache area342ballocated in its RAM302bto store a copy of the data in the local cache area341aof the first CM300a. The first CM300aalso has a mirror cache area342aallocated in its RAM302ato store a copy of the data in the local cache area341bof the second CM300b.

Each time the local cache area341ais changed with new data, the I/O access controller321arequests the I/O access controller321bin the second CM300bto apply the same change to its mirror cache area342b. Similarly, each time the local cache area341bis changed with new data, the I/O access controller321brequests the I/O access controller321ain the first CM300ato apply the same change to its mirror cache area342a.

FIG. 9illustrates access control in solitary operation mode. When the I/O access controller in one CM finds that the RoC device in the other CM has stopped due to some anomalies, the I/O access controller causes the failed RoC device to reboot, and for the time being until the reboot is done, the I/O access controller takes over the task of access control from its peer I/O access controller in the other CM by using the data stored its own mirror cache area.

It is assumed in the example seen inFIG. 9that the RoC device301bin the second CM300bhas stopped its operation. In the illustrated case, the I/O access controller321ain the first CM300acauses the boot controller324ato send a request to its peer boot controller324bin the second CM300bso as to make the RoC device301breboot in restoration mode. The I/O access controller321asubsequently migrates to solitary operation mode, in which the I/O access controller321acontrols access operations, not only to volume Vol#0, but also to volume Vol#1. Specifically, the I/O access controller321auses cache data in the mirror cache area342aof the RAM302ato take over the access control tasks for volume Vol#1.

Suppose that the RoC device301bin the second CM300breboots properly. The I/O access controller321ais now allowed to resume communication with its peer I/O access controller321bin the second CM300b. Via the peer I/O access controller321b, the I/O access controller321aduplexes cache data between the local cache area341aand the mirror cache area342bin the second CM300b, as well as between the mirror cache area342aand the local cache area341bin the second CM300b. The two I/O access controllers321aand321bthen begin their respective tasks of access control in redundant operation mode. However, the local cache area341aand mirror cache area342aof the first CM300amay not experience any changes in their data content during the period of solitary operation mode. When this is the case, the I/O access controller321bin the second CM300bcan restart access control operation to volume Vol#1without updating the local cache area341bor mirror cache area342b.

The I/O access controllers execute access control on the basis of control data defined in a configuration data table as will be discussed inFIG. 10. Each time a change is made to control data in the configuration data table, the configuration data update count is incremented.FIG. 10illustrates how configuration data tables and configuration data update counts are updated.

For example, the first CM300ahas a configuration data table351ain its SSD303a, and the second CM300bhas a configuration data table351bin its SSD303b. These configuration data tables351aand351bcontain control data for use by the I/O access controllers321aand321bto control access to HDDs in the DE200. For example, the configuration data tables351aand351bcontain information that describes RAID levels and defines physical volumes constituting a RAID group. The I/O access controller321aexecutes access control on the basis of such control data defined in the configuration data table351a. Likewise, the I/O access controller321bexecutes access control on the basis of control data defined in the configuration data table351b.

The I/O access controllers321aand321balso manage the configuration data tables351aand351bin the first and second CMs300aand300b, respectively, so as to keep their contents identical. Suppose, for example, a control terminal130issues a request to the first CM300ato make a change to its configuration data table351aas illustrated inFIG. 10. In response, the I/O access controller321ain the first CM300achanges the configuration data table351aas requested by the control terminal130. Subsequently the I/O access controller321aincrements the configuration data update count331astored in the NVRAM309a.

The I/O access controller321afurther interacts with its peer I/O access controller321bin the second CM300bto update the configuration data table351bin the second CM300bwith the same change made to the configuration data table351a, thereby duplexing the table content. In response, the I/O access controller321bin the second CM300bchanges its local configuration data table351bas requested, and then increments the configuration data update count331bstored in the NVRAM309bsimilarly. The configuration data update counts331aand331bin the two CMs300aand300bare thus synchronized with each other.

As can be seen from the above description, the configuration data tables351aand351bin two CMs300aand300bare controlled such that their contents are identical. This feature enables, for example, one of the two CMs to move from redundant operation mode to solitary operation mode and immediately start access control of HDDs by using control data stored in that CM's configuration data table.

FIG. 11illustrates a process of dual CM restoration. When both RoC devices301aand301bin the first and second CMs300aand300bstop their operation at the same time, a dual CM restoration process is invoked to enable each CM300aand300bto quickly restart the task of access control of HDDs, without losing cache data in the first and second CMs300aand300b. Specifically, the restoration controller of each CM invokes a dual CM restoration process when it is found that the RoC devices in both CMs have stopped working.

In the example ofFIG. 11, the dual CM restoration process is controlled by the restoration controller322ain the first CM300a. Specifically, the monitor323akeeps track of whether the RoC device301ain the first CM300ais working properly. When the RoC device301astops, the monitor323aso notifies the restoration controller322a. The monitor323bin the second CM300b, on the other hand, keeps track of whether its local RoC device301bis working properly. Through this monitor323b, the monitor323ain the first CM300acan also see whether the RoC device301bin the second CM300bis working properly. When the RoC device301bstops, the monitor323aso notifies the restoration controller322a.

Based on the notification from the monitor323a, the restoration controller322afinds that both the two RoC devices301aand301bhave stopped. The restoration controller322athen executes a dual CM restoration process to reboot the RoC devices301aand301bin restoration mode. Specifically, the restoration controller322arequests the boot controller324ain the first CM300ato reboot its local RoC device301ain restoration mode. The restoration controller322aalso causes the boot controller324ato communicate with its peer boot controller324bin the second CM300bto request rebooting of its local RoC device301bin restoration mode.

In response to the request from the restoration controller322a, the RoC devices301aand301breboot together and restart their communication. At the time of this rebooting, the RAM302ain the first CM300acontains cache data that has not changed since the RoC device301astopped its operation. Likewise, the RAM302bin the second CM300bcontains cache data that has not changed since the RoC device301bstopped its operation. Accordingly the I/O access controller321acan restart access control operations of volume Vol#0immediately after the RoC device301areboots, by using the cache data maintained in the RAM302a. The I/O access controller321bcan similarly restart access control operations of volume Vol#1immediately after the RoC device301breboots, by using the cache data maintained in the RAM302b.

The above-described dual CM restoration process restores the access control function for HDDs without losing cache data when two RoC devices301aand301bstop together. This restoration process, however, may not work in the case where the two RoC devices stop successively with a time interval in between, because their latest cache data would be lost as will be discussed inFIG. 12.

FIG. 12is a timing diagram illustrating, for reference purposes, an example where two RoC devices stop successively with a certain time interval. Specifically, this example assumes that the RoC device301bin the second CM300bstops first, and then the RoC device301ain the first CM300afollows.

Referring to the initial state seen inFIG. 12, the RoC device301ain the first CM300ais working together with the RoC device301bin the second CM300bin redundant operation mode. The latter RoC device301bthen stops working due to some anomalies (at time T11), and this event is detected by the I/O access controller321ain the first CM300a. Upon detection, the I/O access controller321arequests the boot controller324bin the second CM300bto reboot the RoC device301bin restoration mode (at time T12).

The RoC device301bin the second CM300binitiates a reboot process in restoration mode. While the RoC device301bitself is initialized during this reboot process, the RAM302bis allowed to retain the current data in its local cache area341band mirror cache area342b.

In the first CM300a, on the other hand, the I/O access controller321amakes its way to solitary operation mode, in which the I/O access controller321auses the local cache area341aand mirror cache area342ato continue access control operations for both volumes Vol#0and Vol#1. The I/O access controller321areceives and executes a write request from a host device120to volume Vol#0or volume Vol#1and changes its cache data in the RAM302aaccordingly (at time T13). Since the cache data in the RAM302ais changed, its counterpart in the RAM302bof the second CM300bbecomes obsolete and unusable.

Suppose here that the RoC device301ain the first CM300aalso stops due to some anomalies (at time T14) before the RoC device301bin the second CM300bfinishes its reboot. When the RoC device301bin the second CM300breboots completely, the I/O access controller321bin the second CM300bchecks its connection with the RoC device301ain the first CM300a(at time T15). Since the RoC device301ais in a stop state, the I/O access controller321bfinds it not possible to reach the RoC device301b.

The I/O access controller321bdetermines that the access control should be performed in single operation mode since it is unable to connect with the RoC device301ain the first CM300a. The I/O access controller321bthus requests the boot controller324ain the first CM300ato put the first CM300ainto a fault state (power down state). The I/O access controller321balso invokes an initial setup process in preparation for access control in solitary operation mode (at time T16). For example, the initial setup process is performed such that a mirror cache area342bin the second CM300bcan be used together with a local cache area341b. The initial setup process also configures the I/O access controller321bso as to control both volumes Vol#0and Vol#1. Upon completion of this initial setup process, the I/O access controller321bstarts access control in solitary operation mode (at time T17).

Suppose, for example, that there were no changes to cache data in the RAM302aof the first CM300aduring the period from T12to T14in the above process ofFIG. 12. In that case, the cache data in the RAM302bof the second CM300bcould stay up-to-date even after the rebooting of the RoC device301b. Accordingly, the RoC device301bin the second CM300bwould be able to execute access control properly in solitary operation mode by using the cache data in the local cache area341band mirror cache area342bof the RAM302bat time T17.

However, the cache data in the RAM302bof the second CM300bloses its up-to-dateness because a change is made to cache data in the first CM300aduring the period of T12to T14as depicted inFIG. 12. If the RoC device301ain the first CM300astops in this condition (at time T14), the I/O access controller321bin the second CM300bthen starts access control in solitary operation mode (at time T17). The latest cache data in the RAM302ain the first CM300ais lost because the RoC device301ais not operational.

In another hypothetical case, a write request to HDDs in the DE200arrives during the period between T12and T14, but the RoC device301ain the first CM300astops working before the I/O access controller321aexecutes the write request. In this case, the specified write data would be lost. The failed data write operation necessitates extra work to be done by a system administrator before resuming the access control. More specifically, the system administrator has to search the event log of each CM300aand300bto track down the failed events and recover the lost write data. It thus takes a long time to restart access control operation.

To overcome the problems discussed above, the storage system100according to the second embodiment is designed such that a first CM causes a second CM to reboot in restoration mode, not to lose the latest cache data in the second CM, in the case where the first CM's RoC device rebooted on request from the second CM finds itself unable to connect with the second CM.

FIGS. 13 and 14are a timing diagram illustrating how the second embodiment works when two RoC devices stop successively with a certain interval. Specifically, this example assumes that the RoC device301bin the second CM300bstops first, and then the RoC device301ain the first CM300afollows.

Referring to the initial state seen inFIG. 13, the RoC device301ain the first CM300ais working together with the RoC device301bin the second CM300bin redundant operation mode. The latter RoC device301bthen stops working due to some anomalies (at time T21), and this event is detected by the I/O access controller321ain the first CM300a(at time T22). Upon detection, the I/O access controller321arequests the second CM300bvia the boot controller324ato change its boot flag332bin the NVRAM309bfrom zero to one. In addition, the I/O access controller321arequests the boot controller324bin the second CM300bto reboot the stopped RoC device301bin restoration mode.

The I/O access controller321afurther increments the configuration data update count331astored in the NVRAM309aof the first CM300a. As discussed inFIG. 10, the configuration data update count331ain the first CM300ahas the same value as the configuration data update count331bin the second CM300b, as long as both RoC devices301aand301bare operating properly. At the time point T22, however, the RoC device301bin the second CM300bis in a stop state, being unable to increment its configuration data update count331b. Only the configuration data update count331ain the first CM300ais incremented in this situation.

In response to the request from the I/O access controller321ain the first CM300a, the RoC device301bin the second CM300bstarts rebooting in restoration mode. In this rebooting process, the power supply circuit310btemporarily stops supplying power to the RoC device301bso as to reset its circuits. After that, the RoC device301bloads necessary programs from the SSD303aand runs them to get ready to execute a connection check as the first step to do. The current data in the local cache area341band mirror cache area342bof RAM302bis kept intact during this period of rebooting the RoC device301b.

In the first CM300a, on the other hand, the I/O access controller321amakes its way to solitary operation mode, in which the I/O access controller321auses the local cache area341aand mirror cache area342ato continue access control operations for both volumes Vol#0and Vol#l. The I/O access controller321anow receives and executes a write request from a host device120to volume Vol#0or volume Vol#1and changes its cache data in the RAM302aaccordingly (at time T23).

Suppose here that the RoC device301ain the first CM300astops due to some anomalies (at time T24) before the RoC device301bin the second CM300bfinishes its reboot. When the RoC device301bin the second CM300breboots completely, the I/O access controller321bin the second CM300bchecks its connection with the RoC device301ain the first CM300a(at time T25). Since the RoC device301ais stopped, the I/O access controller321bfinds it not possible to reach the RoC device301a.

Since the peer RoC device301acannot be reached, the I/O access controller321bchooses and executes an appropriate process depending on the boot flag332b, which is stored in the NVRAM309bof the second CM300b. Specifically, when the boot flag332bis set to one, the I/O access controller321bstops the RoC device301bin the second CM300b(at time T26).

While not illustrated inFIG. 13, the I/O access controller321bis supposed to execute an initial setup process for access control in solitary operation mode if the peer RoC device301acannot be reached at time T25, and if the boot flag332bis zero. After this initial setup process, the I/O access controller321bcontrols access to volumes Vol#0and Vol#1by itself.

Referring again toFIG. 13, the RoC device301bin the second CM300bstops its operation at time T26. The restoration controller322ain the first CM300adetects that both RoC devices301aand301bhave stopped, as illustrated inFIG. 14. The restoration controller322bin the second CM300bmay also detect the event. In the former case, for example, the restoration controller322aexecutes a dual CM restoration process to reboot both RoC devices301aand301bin restoration mode (at time T27inFIG. 14). Specifically, the restoration controller322arequests the boot controller324ain the first CM300ato reboot its local RoC device301ain restoration mode. The restoration controller322aalso causes the boot controller324ato request its peer boot controller324bin the second CM300bto reboot the RoC device301bin restoration mode.

In response to the request from the restoration controller322a, the RoC device301ain the first CM300areboots while keeping its own cache data intact, as does the RoC device301bin the second CM300b. When both RoC devices301aand301bcomplete their respective reboot processes, the I/O access controllers321aand321bcheck their connection over communication path P1(at time T28). Here the I/O access controllers321aand321balso compare the configuration data update count331ain NVRAM309aof the first CM300awith the configuration data update count331bin NVRAM309bof the second CM300b. Because the configuration data update count331aof the first CM300ais greater than the configuration data update count331bof the second CM300b, the I/O access controllers321aand321brecognize the validity (i.e., up-to-dateness) of cache data stored in the first CM300aand execute an initial setup process in preparation for access control in redundant operation mode (at time T29). The configuration data update count331bin the second CM300bis incremented at the start of this initial setup process, which renders the two configuration data update counts331aand331bin a synchronized state.

During the course of the initial setup process started at time T29, the I/O access controllers321aand321bexecute a task of duplexing cache data held in the first CM300a. More specifically, the I/O access controller321asends data from its own local cache area341aand mirror cache area342ain the RAM302ato the peer I/O access controller321bin the second CM300b. The I/O access controller321bdiscards data in the local cache area341band mirror cache area342bof the RAM302b, and loads the mirror cache area342bwith new data sent from the local cache area341a, as well as the local cache area341bwith new data sent from the mirror cache area342a. The I/O access controllers321aand321bstart, upon completion of their respective initial setup processes, access control operations in redundant operation mode (at time T30).

According to the above-described process of FIGS. and14, the boot flag332benables the I/O access controller321bin the second CM300bto determine, at the time of its rebooting, whether the rebooting has been initiated by the first CM300a. When it is determined that the rebooting has been initiated by the first CM300a, and if the RoC device301ain that first CM300ais then in a stop state, the RoC device301bin the second CM300bstops itself to intentionally create a situation that necessitates a dual CM restoration process. By so doing, the RoC device301bmakes the peer RoC device301areboot with valid cache data maintained therein. This action enables the RoC devices301aand301bto restart access control over HDDs in the DE200by using the valid cache data maintained in the RoC device301a.

The above-described processing makes it possible to restart the task of access control without losing the latest cache data even in the case where the RoC devices301aand301bin the first and second CMs300aand300bsuccessively stop working with a certain time interval. The above-described processing also enables quick and automated restoration of access control functions without intervention of the system administrator even in the noted case.

The I/O access controllers321aand321bin the first and second CMs300aand300bcheck their connection at time T28as a result of the dual CM restoration process, and at that point, the I/O access controllers321aand321bdetermine which of the first and second CMs300aand300bhas valid cache data. It is noted that the I/O access controllers321aand321bdo not require any additional information for this determination, but can achieve it by comparing their configuration data update counts331aand331b, which are originally used for other purposes.

The configuration data tables351aand351bmay contain log records of anomalous events, such as a stop state of RoC devices301aand301b. In this case, the I/O access controller321ain the first CM300amay take care of such error event records in the configuration data table351a. For example, when the RoC device301bin the second CM300bstops at time T22ofFIG. 13, the I/O access controller321aupdates its configuration data table351ato record that event. The I/O access controller321aalso increments its configuration data update count331awhen such an update takes place in the configuration data table351a. After that, the I/O access controller321aduplexes the configuration data table351aby reflecting new content in the configuration data table351bof the second CM300bwhen the RoC device301ain the first CM300areboots and starts an initial setup for access control (as in time T29ofFIG. 14, or as in step S35ofFIG. 17to be described later).

The operation of CMs according to the second embodiment will now be described with reference to some flowcharts for the first CM300a. The described operation may also be executed in a similar way by the second CM300b.

FIG. 15is a flowchart of a process executed by an I/O access controller to supervise the activity of its peer RoC device. Specifically, the illustrated process ofFIG. 15is executed by the I/O access controller321ain the first CM300awhen it performs access control in redundant operation mode.

(Step S11) The I/O access controller321amonitors whether the RoC device301bin the second CM300bis running or stopped. For example, the I/O access controller321aachieves this by communicating health-check signals with the RoC device301bin the second CM300bover communication path P1at regular intervals. The I/O access controller321amoves to step S12when it is determined that the RoC device301bis stopped (Yes at step S11).

(Step S12) Through the boot controller324a, the I/O access controller321arequests the boot controller324bin the second CM300bto change its boot flag332bfrom zero to one. In response to this request, the boot controller324bin the second CM300bchanges its boot flag332bin the NVRAM309bto one.

(Step S13) Through the boot controller324a, the I/O access controller321arequests the boot controller324bin the second CM300bto reboot the RoC device301bin restoration mode. In response to this request, the boot controller324bmakes the RoC device301breboot by interrupting its power supply for a short time to reset the RoC device301bwhile keeping the RAM302bpowered.

(Step S14) The I/O access controller321aincrements the configuration data update count331ain NVRAM309aof the first CM300a.

FIG. 16is a flowchart of a process executed by the restoration controller.

(Step S21) In the first CM300a, the monitor323akeeps track of whether the RoC device301ais working properly. When the RoC device301astops, the monitor323aso notifies the restoration controller322a. The monitor323bin the second CM300b, on the other hand, keeps track of whether the RoC device301bis working properly. Through this monitor323b, the monitor323ain the first CM300acan see whether the RoC device301bin the second CM300bis working properly. When the RoC device301bstops, the monitor323aso notifies the restoration controller322a.

The restoration controller322achecks whether there is a notification from the monitor323awhich indicates a stop state of each RoC device301aand301b. When the monitor323aindicates that both RoC devices301aand301bhave stopped (Yes at step S21), the restoration controller322aproceeds to step S22.

(Step S22) The restoration controller322aexecutes a dual CM restoration process. Specifically, the restoration controller322arequests the boot controller324ato reboot its local RoC device301ain restoration mode. The restoration controller322aalso requests, via the boot controller324a, the boot controller324bin the second CM300bto reboot its local RoC device301bin restoration mode. The two RoC devices301aand301bthus reboot together in response to the request from the restoration controller322a, while keeping their respective cache data intact.

FIGS. 17 and 18give a flowchart illustrating what is executed when an RoC device starts up. The illustrated process ofFIGS. 17 and 18is executed when the first CM300astarts upon power up, as well as when the first CM300areboots at its own discretion or upon request from the second CM300b.

(Step S31) The RoC device301aruns a boot process. During this course, the hardware of the RoC device301ais initialized, access control programs are loaded from the SSD303ainto the RoC device301a, and other necessary processing is done. The CPU in the RoC device301athen begins executing the access control programs, thereby launching an I/O access controller321a.

(Step S32) The I/O access controller321atests its connection with the peer RoC device301bvia communication path P1. When the RoC device301bcan be reached (Yes at step S32), the I/O access controller321aproceeds to step S33. When the RoC device301bcannot be reached (No at step S32), the I/O access controller321aproceeds to step S39.

(Step S33) Through the boot controller324a, the I/O access controller321areads a configuration data update count331aout of the NVRAM309a. In addition, the I/O access controller321ainteracts with its peer I/O access controller321bin the second CM300bto read a configuration data update count331bstored in the NVRAM309bof the second CM300b.

The I/O access controller321acompares the read values of configuration data update counts331aand331b. If the two values match with each other, the I/O access controller321aproceeds to step S34. If the configuration data update count331ain the first CM300ais greater than the configuration data update count331bread out of the second CM300b, the I/O access controller321aproceeds to step S35. If the configuration data update count331ain the first CM300ais smaller than the configuration data update count331bread out of the second CM300b, the I/O access controller321aproceeds to step S37.

(Step S34) The I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. It is noted that this initial setup process involves no data transfer or update concerning the local cache area341aand mirror cache area342aof the RAM302a. Upon completion of the initial setup process, the I/O access controller321astarts access control operations in redundant operation mode.

This step S34is executed in the case where, for example, the RoC device301areboots as a result of a dual CM restoration process that is initiated because of simultaneous stop of RoC devices301aand301bduring access control operations by the I/O access controllers321aand321bin redundant operation mode. In this particular case, both CMs300aand300bhave the latest cache data, which permits the I/O access controller321ato resume access control operations in redundant operation mode without the need for reestablishing the duplexed state of cache data.

(Step S35) Now that the configuration data update count331ain the first CM300ais greater than the configuration data update count331bin the second CM300b, the I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. During the course of this initial setup process, the I/O access controller321arequests the I/O access controller321bin the second CM300bto update its local cache data, sending the current content of the local cache area341aand mirror cache area342a. In response, the I/O access controller321bin the second CM300bdiscards its cache data in the RAM302band then loads the mirror cache area342bof the second CM300bwith the data from the local cache area341a, as well as the local cache area341bof the second CM300bwith the data from the mirror cache area342a.

(Step S36) During the course of the above initial setup process, the I/O access controller321arequests the I/O access controller321bin the second CM300bto equalize its configuration data update count331bto the configuration data update count331ain the first CM300a. In response, the I/O access controller321bexecutes the request by, for example, incrementing the configuration data update count331bin the NVRAM309b.

Upon completion of the above initial setup process, the I/O access controller321astarts access control operations in redundant operation mode. The above-described series of steps S31to S33, S35, and S36may correspond to, for example, the foregoing process that the first CM300aexecutes after rebooting at time T27inFIG. 14.

(Step S37) The I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. During the course of this initial setup process, the I/O access controller321aflushes its cache data in the RAM302aand then receives replacement data from the peer I/O access controller321bin the second CM300b. This data is what is stored in of the local cache area341band mirror cache area342bof the second CM300b. The I/O access controller321astores the received data of the local cache area341band mirror cache area342bin the mirror cache area342aand local cache area341a, respectively.

(Step S38) During the course of the above initial setup process, the I/O access controller321areceives a request from its peer I/O access controller321bin the second CM300bfor updating the configuration data update count331ain the first CM300a. In response, the I/O access controller321aexecutes the request by, for example, incrementing the configuration data update count331ain the NVRAM309aso as to equalize it to the configuration data update count331bin the second CM300b.

Upon completion of the above initial setup process, the I/O access controller321astarts access control operations in redundant operation mode. The above-described series of steps S31to S33, S37, and S38may correspond to, for example, the foregoing process that the second CM300bexecutes after rebooting at time T27inFIG. 14.

(Step S39) Since the peer RoC device301bcannot be reached, the I/O access controller321aconsults its own boot flag332ain the NVRAM309a. If the boot flag332ais set to one, I/O access controller321aproceeds to step S40. If the boot flag332ais zero, the I/O access controller321aproceeds to step S42.

(Step S41) The I/O access controller321aforces the RoC device301ato stop its operation, thus intentionally creating a situation as if the RoC device301ahad stopped due to some anomalies.

The above-described series of steps S31, S32, S39to S41may correspond to, for example, the foregoing process that the second CM300bstarts after rebooting at time T22inFIG. 13and continues until the RoC device301bis stopped at time T26. Accordingly, the a dual CM restoration process is supposed to be invoked after step S41, which causes the RoC device301ato reboot in restoration mode.

(Step S42) Now that the boot flag332ais found to be zero, the I/O access controller321aexecutes an initial setup process in preparation for access control in solitary operation mode. For example, this initial setup process is performed such that the current data in the local cache area341aand mirror cache area342ain the second CM300bcan be used as cache data. The initial setup process also configures the I/O access controller321aso as to control both volumes Vol#0and Vol#1. Upon completion of the initial setup process, the I/O access controller321astarts access control operations for both volumes Vol#0and Vol#1in solitary operation mode.

The above-described series of steps S31, S32, S39, and S42may correspond to, for example, the process executed in the case where the RoC device301ain the first CM300acannot reboot properly, while the RoC device301bin the second CM300bcan, in the dual CM restoration process initiated at time T27inFIG. 14. In this case, the I/O access controller321bin the second CM300bis unable to reach the RoC device301ain the first CM300a(No at step S32), and the boot flag332bis zero (step S39). Accordingly, the I/O access controller321bbegins access control in solitary operation mode upon completion of an initial setup process therefor (step S42).

In the foregoing procedure ofFIG. 14, two RoC devices301aand301bconfirm their connection at time T28, and the RoC device301bin the second CM300bis caused to start an initial setup process at time T29because of its lack of valid cache data. As an alternative procedure, the I/O access controller321ain the first CM300amay be configured to cause its peer RoC device301bto reboot in normal mode at time T29. In this case, the RoC device301bdiscards cache data in the RAM302bwhen it reboots. The I/O access controller321amay control access to volume Vol#0and Vol#1in solitary operation mode until the RoC device301breboots completely and becomes reachable again. After confirming its connection with the rebooted RoC device301b, the I/O access controller321aprovides a copy of cache data in its local RAM302ato the RAM302bin the second CM300bduring the course of an initial setup process of the I/O access controller321b, thereby establishing a duplexed state of cache data. When the I/O access controller321bcompletes its initial setup process, the two I/O access controllers321aand321bstart access control together in redundant operation mode.

As another alternative procedure, the I/O access controller321ain the first CM300amay be configured to execute access control of volumes Vol#0and Vol#1in solitary operation mode, while bringing the second CM300bto, for example, a fault state (power down state) at time T29. In this case, the flowchart ofFIG. 17is to be modified such that the I/O access controller renders its peer RoC device into a fault state at step S35, instead of duplexing cache data, when step S33finds that the local CM's configuration data update count is greater than the peer CM's configuration data update count. Step S36may be omitted. Also, when step S33finds that the local CM's configuration data update count is smaller than the peer CM's configuration data update count, the I/O access controller does not execute steps S37and S38, but waits the peer CM to request transition to the fault state.

It is noted that, in the case of the second alternative (i.e., where the second CM300bis rendered into a fault state at time T29inFIG. 14because of its smaller configuration data update count), the I/O access controller321ain the first CM300ais allowed to skip the step of duplexing cache data during its initial setup process at time T29. This means that the first CM300acan resume access control operations in a shorter time.

According to the second embodiment described above, the I/O access controller in a CM causes the RoC device in its peer CM to reboot in restoration mode when that RoC device is found stopped. The I/O access controller may, however, be configured to make the RoC device reboot in normal mode without retaining its cache data. This alternative can be applied to the I/O access controller321ain the first CM300aat time T22in the process ofFIG. 13. In this case, the RoC device301bin the second CM300bis rebooted in normal mode, during which the cache data is lost from the RAM302bin the second CM300b. In other words, the cache data in the second CM300bis invalidated, no matter whether its counterpart in the first CM300ais changed during the period from T22to T24. Accordingly the two CMs300aand300bcan execute subsequent processing after T24similarly to the sequence ofFIG. 13, thereby resuming their access control operations with the latest cache data in the first CM300a.

This section will describe a storage system according to a third embodiment. The third embodiment is different from the foregoing second embodiment in that the first and second CMs300aand300bhave a function of counting changes made to cache data. According to the third embodiment, the I/O access controllers321aand321bin the first and second CMs300aand300bcompare their respective cache change counts when they are rebooted in a dual CM restoration process. Depending on the comparison result, the I/O access controllers321aand321bdetermine whether to duplex cache data or to keep their current cache data intact, before they start access control.

FIG. 19illustrates how cache change counts are incremented. Specifically, a first cache change count343aand a second cache change count344aare located in the NVRAM309aof the first CM300a. The first cache change count343aindicates the number of changes made to cache data in the local cache area341a, and the second cache change count344aindicates the same for the mirror cache area342a. Similarly, the NVRAM309bin the second CM300bstores a first cache change count343band a second cache change count344b. The first cache change count343bindicates the number of changes made to cache data in the local cache area341b, and the second cache change count344bindicates the same for the mirror cache area342b.

When they execute access control in redundant operation mode, the two I/O access controllers321aand321bin the first and second CMs300aand300bcontrol their cache data and associated cache change counts such that a pair of cache change counts343aand343bhave equal values, and so do another pair of cache change counts344aand344b.

More specifically, the I/O access controller321ain the first CM300aincrements its first cache change count343awhen cache data in the local cache area341ais changed by, for example, a data write operation. Each time a change is made to the local cache area341a, the I/O access controller321asends the changed data to its peer I/O access controller321bin the second CM300b, so that the same change is applied to the mirror cache area342bin the second CM300b. The I/O access controller321aalso requests, each time the first cache change count343ais incremented, the I/O access controller321bin the second CM300bto equalize its second cache change count344bto the first cache change count343a. The requested I/O access controller321bsynchronizes the cache change counts by, for example, incrementing the second cache change count344b, or overwriting the second cache change count344bwith the value of first cache change count343asupplied from the peer I/O access controller321a.

In response to the above request from the peer I/O access controller321ain the first CM300a, the I/O access controller321bupdates its own mirror cache area342bwith cache data received from the I/O access controller321a. The I/O access controller321bthen increments the second cache change count344b, thereby equalizing it to the first cache change count343a.

Similarly to the above, the I/O access controller321bin the second CM300bincrements its first cache change count343bwhen cache data in the local cache area341bis changed by, for example, a data write operation. Each time a change is made to the local cache area341b, I/O access controller321bsends the changed data to its peer I/O access controller321ain the first CM300a, so that the same change is applied to the mirror cache area342ain the first CM300a. The I/O access controller321balso requests, each time the first cache change count343bis incremented, the I/O access controller321ain the first CM300ato equalize its second cache change count344ato the first cache change count343b.

In response to the above request from the peer I/O access controller321bin the second CM300b, the I/O access controller321aupdates its own mirror cache area342awith cache data received from the I/O access controller321b. The I/O access controller321athen increments the second cache change count344a, thereby equalizing it to the first cache change count343b.

FIG. 20illustrates how cache change counts are incremented in solitary operation mode. As described above, the I/O access controller in one CM makes the RoC device in the other CM in restoration mode when that RoC device stops due to some anomalies. By using cache data stored in the mirror cache area of the CM, the I/O access controller then takes over the access control tasks that have been done by its counterpart in the other CM. This is continued until the RoC device recovers in the other CM.

Specifically,FIG. 20illustrates an example case where the RoC device301bin the second CM300bis stopped. In the illustrated case, the I/O access controller321ain the first CM300amigrates to solitary operation mode, while making the RoC device301bin the second CM300breboot in restoration mode. In solitary operation mode, the I/O access controller321ais supposed to control access operations, not only to volume Vol#0, but also to volume Vol#l. For the latter purpose, the I/O access controller321auses cache data in the mirror cache area342aof the RAM302a.

In solitary operation mode, the I/O access controller321aincrements its first cache change count343aeach time a change is made to cache data in the local cache area341a. The I/O access controller321aalso increments its second cache change count344aeach time a change is made to cache data in the mirror cache area342a.

FIGS. 21 and 22give a timing diagram illustrating how the third embodiment works when two RoC devices stop successively with a certain interval. Specifically, this example assumes that the RoC device301bin the second CM300bstops first, and then the RoC device301ain the first CM300afollows.

Referring to the initial state seen inFIG. 21, the RoC device301ain the first CM300ais working together with the RoC device301bin the second CM300bin redundant operation mode. The latter RoC device301bthen stops working due to some anomalies (at time T41), and this event is detected by the I/O access controller321ain the first CM300a(at time T42). Upon detection, the I/O access controller321amakes the RoC device301bin the second CM300breboot in restoration mode. The I/O access controller321aalso causes the second CM300bto change its boot flag332bin the NVRAM309bfrom zero to one, as well as incrementing the configuration data update count331astored in the NVRAM309aof the first CM300a.

In response to the above request from the I/O access controller321ain the first CM300a, the RoC device301bin the second CM300bstarts rebooting in restoration mode. During this reboot process, the RAM302bin the second CM300bis allowed to retain the current data in its local cache area341band mirror cache area342b.

In the first CM300a, on the other hand, the I/O access controller321amakes its way to solitary operation mode, in which the I/O access controller321auses the local cache area341aand mirror cache area342ato continue access control operations for both volumes Vol#0and Vol#1. The I/O access controller321anow receives a write request from a host device120to volume Vol#0or volume Vol#1. In response, the I/O access controller321aupdates cache data in the RAM302aand increments a cache change count that corresponds to the updated cache area accordingly (at time T43). At this moment, the I/O access controller321ais unable to communicate with the RoC device301bthat is rebooting in the second CM300b. Thus the I/O access controller321aonly increments its own cache change counts in the first CM300awhile leaving those in the second CM300bas they are.

Suppose here that the RoC device301ain the first CM300astops due to some anomalies (at time T44) before the RoC device301bin the second CM300bfinishes its reboot. When the RoC device301bin the second CM300breboots completely, the I/O access controller321bin the second CM300bchecks its connection with the RoC device301ain the first CM300a(at time T45). Since the RoC device301ais stopped, the I/O access controller321bfinds it not possible to reach the RoC device301a.

The I/O access controller321boperates similarly to the foregoing second embodiment when the peer RoC device301acannot be reached. Specifically, the I/O access controller321bchooses and executes an appropriate process depending on the boot flag332bin NVRAM309bof the second CM300b. If the boot flag332bis one, the I/O access controller321bstops the RoC device301bin the second CM300b(at time T46).

The RoC device301bin the second CM300bthus stops its operation at time T46. The restoration controller322ain the first CM300adetects that both RoC devices301aand301bhave stopped, as illustrated inFIG. 22. The restoration controller322bin the second CM300bmay also detect the event. In the former case, for example, the restoration controller322aexecutes a dual CM restoration process to reboot both RoC devices301aand301bin restoration mode (at time T47). The restoration controller322amakes its local RoC device301a, as well as the RoC device301bin the second CM300b, reboot in restoration mode.

When both RoC devices301aand301bcomplete their respective reboot processes, the I/O access controllers321aand321bin the first and second CMs300aand300bsuccessfully confirm their connection over communication path P1(at time T48). Here the I/O access controllers321aand321bcompares the configuration data update count331ain NVRAM309aof the first CM300awith the configuration data update count331bin NVRAM309bof the second CM300b, as in the foregoing second embodiment.

If the configuration data update count331aof the first CM300ais greater than the configuration data update count331bof the second CM300b, then the I/O access controllers321aand321bcompare cache change counts stored in NVRAM309aof the first CM300awith those stored in NVRAM309bof the second CM300b. If the first cache change count343ais greater than the second cache change count344b, or if the second cache change count344ais greater than the first cache change count343b, or if both of those conditions are met, then the I/O access controllers321aand321bresume their access control in redundant operation mode after duplexing cache data as illustrated inFIG. 14. According to the present embodiment, the duplexing of cache data has only to operate on the cache areas whose cache change counts do not coincide.

Referring again toFIG. 21, when no changes are made to cache data in the first CM300aduring the period of T42to T44, the cache change counts stored in the first CM300acoincide with those stored in the second CM300bat time T48. If the second cache change count344bcoincides with the first cache change count343a, and if the first cache change count343bcoincides with the second cache change count344a, the I/O access controllers321aand321bresume access control in redundant operation mode without duplexing cache data. This means that the two CMs300aand300bcan resume their access control operations in a shorter time than in the second embodiment, in the case where no changes are made to cache data in the first CM300aduring the period of T42to T44inFIG. 21.

FIG. 23is a flowchart illustrating what is executed when an RoC device starts up according to the third embodiment. The boot process of RoC devices in the third embodiment shares some steps with the process discussed inFIGS. 17 and 18for the second embodiment, but is different from the following points. That is, steps S35and S36have been replaced with steps S51to S53inFIG. 23. Steps S37and S38have been replaced with steps S54to S56inFIG. 23. The following description ofFIG. 23will focus on these modified steps.

(Step S51) When the I/O access controller321ain the first CM300acan reach its peer RoC device301bin the second CM300b(Yes at step S32), and when the configuration data update count331aof the first CM300ais greater than the configuration data update count331bof the second CM300b(step S33), the I/O access controller321acompares cache change counts stored in the first CM300awith those stored in the second CM300b.

Specifically, the I/O access controller321areads first and second cache change counts343aand344aout of the NVRAM309avia the boot controller324a. The I/O access controller321aalso interacts with its peer I/O access controller321bin the second CM300bto read first and second cache change counts343band344bout of the NVRAM309bin the second CM300b. The I/O access controller321aproceeds to step S52, if the first cache change count343ais greater than the second cache change count344b, or if the second cache change count344ais greater than the first cache change count343b, or if both of those conditions are met. The I/O access controller321aproceeds to step S34, if the second cache change count344bequals the first cache change count343a, and if the first cache change count343bequals the second cache change count344a. At step S34, the I/O access controller321aexecutes an initial setup process for access control, which does not include duplexing of cache data.

(Step S52) The I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. During the course of this initial setup process, the I/O access controller321aupdates cache data in the local cache area341band mirror cache area342bof the second CM300bwith its own cache data stored in the mirror cache area342aand local cache area341a, respectively.

When the comparison at step S51has revealed that the first cache change count343ais greater than the second cache change count344b, the I/O access controller321asends data from the local cache area341ato the I/O access controller321bin the second CM300band requests the I/O access controller321bto update the mirror cache area342bin the second CM300b. In response, the I/O access controller321bin the second CM300bdiscards its cache data in the mirror cache area342band then stores the received cache data in the emptied mirror cache area342b.

When the comparison at step S51has revealed that the second cache change count344ais greater than the first cache change count343b, the I/O access controller321asends data from the mirror cache area342ato the I/O access controller321bin the second CM300band requests the I/O access controller321bto update the local cache area341bin the second CM300b. In response, the I/O access controller321bdiscards its cache data in the local cache area341band then stores the received cache data in the local cache area341b.

(Step S53) During the course of the above initial setup process, the I/O access controller321arequests the I/O access controller321bin the second CM300bto equalize its configuration data update count331bto the configuration data update count331ain the first CM300a. In response, the I/O access controller321bexecutes the request by, for example, incrementing the configuration data update count331bin the NVRAM309b.

The I/O access controller321afurther requests its peer I/O access controller321bto update cache change counts in the second CM300b. Specifically, when the comparison at step S51has revealed that the first cache change count343ais greater than the second cache change count344b, the I/O access controller321asends that first cache change count343ato its peer I/O access controller321bin the second CM300b, thus requesting update of the second cache change count344b. In response, the I/O access controller321bupdates its second cache change count344bwith the received first cache change count343a.

Similarly, when the comparison at step S51has revealed that the second cache change count344ais greater than the first cache change count343b, the I/O access controller321asends that second cache change count344ato its peer I/O access controller321bin the second CM300b, thus requesting update of the first cache change count343b. In response, the I/O access controller321bupdates the first cache change count343bwith the received second cache change count344a.

Upon completion of the above initial setup process, the I/O access controller321aresumes access control operations in redundant operation mode. The above-described series of steps S31to S33and S51to S53may correspond to, for example, the foregoing process that the first CM300aexecutes after it is rebooted at time T47inFIG. 22and finds that the RoC device301bin the second CM300bis reachable.

(Step S54) When the I/O access controller321ain the first CM300acan reach its peer RoC device301bin the second CM300b(Yes at step S32), and when the configuration data update count331aof the first CM300ais smaller than the configuration data update count331bof the second CM300b(step S33), the I/O access controller321acompares cache change counts stored in the first CM300awith those stored in the second CM300b.

Specifically, the I/O access controller321areads first and second cache change counts343aand344aout of the NVRAM309avia the boot controller324a. The I/O access controller321aalso interacts with its peer I/O access controller321bin the second CM300bto read first and second cache change counts343band344bout of the NVRAM309bin the second CM300b. The I/O access controller321aproceeds to step S55, if the first cache change count343ais smaller than the second cache change count344b, or if the second cache change count344ais smaller than the first cache change count343b, or if both of those conditions are met.

The I/O access controller321a, on the other hand, proceeds to step S34, if the second cache change count344bequals the first cache change count343a, and if the first cache change count343bequals the second cache change count344a. At step S34, the I/O access controller321aexecutes an initial setup process for access control, which does not include duplexing of cache data.

(Step S55) The I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. During the course of this initial setup process, the I/O access controller321aupdates at least one of the local cache area341aand mirror cache area342aof the RAM302awith cache data sent from the I/O access controller321bin the second CM300b.

Specifically, when the comparison at step S54has revealed that the first cache change count343ais smaller than the second cache change count344b, the I/O access controller321adiscards its cache data in the local cache area341aand then loads the emptied local cache area341awith cache data of the mirror cache area342bwhich is received from the peer I/O access controller321b. Likewise, when the comparison at step S54has revealed that the second cache change count344ais smaller than the first cache change count343b, the I/O access controller321adiscards its cache data in the mirror cache area342aand then loads the emptied mirror cache area342awith cache data of the local cache area341bwhich is received from the peer I/O access controller321b.

(Step S56) During the course of the above initial setup process, the I/O access controller321areceives a request from its peer I/O access controller321bin the second CM300bfor updating the configuration data update count331ain the first CM300a. In response, the I/O access controller321aexecutes the request by, for example, incrementing the configuration data update count331ain the NVRAM309aso as to equalize it to the configuration data update count331b.

The I/O access controller321afurther updates at least one of the first and second cache change counts343aand344astored in the NVRAM309a. Specifically, when the comparison at step S54has revealed that the first cache change count343ais smaller than the second cache change count344b, the I/O access controller321aupdates the first cache change count343awith the value of the second cache change count344breceived from its peer I/O access controller321bin the second CM300b. Likewise, when the comparison at step S54has revealed that the second cache change count344ais smaller than the first cache change count343b, the I/O access controller321aupdates the second cache change count344awith the value of the first cache change count343breceived from its peer I/O access controller321bin the second CM300b.

Upon completion of the above initial setup process, the I/O access controller321aresumes access control operations in redundant operation mode. The above-described series of steps S31to S33and S54to S56may correspond to, for example, the foregoing process that the second CM300bexecutes after it reboots at time T47inFIG. 22and finds that the RoC device301ain the first CM300ais reachable.

The third embodiment described above relies on configuration data update counts in addition to cache change counts to determine what to do before starting access control operations. By contrast, according to the fourth embodiment described below, the I/O access controller in a booted CM determines the same by consulting cache change counts, but not configuration data update counts.

FIG. 24is a flowchart illustrating what is executed when an RoC device starts up according to the fourth embodiment. The present embodiment executes, when an RoC device boots up, a process similar to that of FIGS.17and18, except that steps S33, S35, S36, S37, and S38are respectively replaced with steps S61, S62, S63, S64, and S65as seen inFIG. 24. The following description ofFIG. 24will focus on these modified steps.

(Step S61) When the I/O access controller321ain the first CM300acan reach its peer RoC device301bin the second CM300b(Yes at step S32), the I/O access controller321acompares cache change counts stored in the first CM300awith those stored in the second CM300b.

Specifically, the I/O access controller321areads first and second cache change counts343aand344aout of the NVRAM309avia the boot controller324a. The I/O access controller321aalso interacts with its peer I/O access controller321bin the second CM300bto read first and second cache change counts343band344bout of the NVRAM309bin the second CM300b. The I/O access controller321aproceeds to step S62if the first cache change count343ais greater than the second cache change count344b, or if the second cache change count344ais greater than the first cache change count343b, or if both of those conditions are met. The I/O access controller321aproceeds to step S64if the first cache change count343ais smaller than the second cache change count344b, or if the second cache change count344ais smaller than the first cache change count343b, or if both of those conditions are met.

The I/O access controller321a, on the other hand, proceeds to step S34, if the second cache change count344bequals the first cache change count343a, and if the first cache change count343bequals the second cache change count344a. At step S34, the I/O access controller321aexecutes an initial setup process for access control, without duplexing cache data.

(Step S62) The I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. During the course of this initial setup process, the I/O access controller321aupdates cache data in the local cache area341band mirror cache area342bof the second CM300bwith its own cache data stored in the mirror cache area342aand local cache area341a, respectively.

Specifically, when the comparison at step S61has revealed that the first cache change count343ais greater than the second cache change count344b, the I/O access controller321asends data from the local cache area341ato the I/O access controller321bin the second CM300band requests the I/O access controller321bto update the mirror cache area342bin the second CM300b. In response, the I/O access controller321bin the second CM300bdiscards its cache data in the mirror cache area342band then stores the received cache data in the emptied mirror cache area342b.

When the comparison at step S61has revealed that the second cache change count344ais greater than the first cache change count343b, the I/O access controller321asends data from the mirror cache area342ato the I/O access controller321bin the second CM300band requests the I/O access controller321bto update the local cache area341bin the second CM300b. In response, the I/O access controller321bdiscards its cache data in the local cache area341band then stores the received cache data in the emptied local cache area341b.

(Step S63) During the course of the above initial setup process, the I/O access controller321arequests the I/O access controller321bin the second CM300bto update its cache change counts. Specifically, when the comparison at step S61has revealed that the first cache change count343ais greater than the second cache change count344b, the I/O access controller321asends that first cache change count343ato its peer I/O access controller321bin the second CM300b, thus requesting update of the second cache change count344b. In response, the I/O access controller321bupdates its second cache change count344bwith the received first cache change count343a.

When the comparison at step S61has revealed that the second cache change count344ais greater than the first cache change count343b, the I/O access controller321asends that second cache change count344ato its peer I/O access controller321bin the second CM300b, thus requesting update of the first cache change count343b. In response, the I/O access controller321bupdates the first cache change count343bwith the received second cache change count344a.

Upon completion of the above initial setup process, the I/O access controller321aresumes access control operations in redundant operation mode.

(Step S64) The I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. During the course of this initial setup process, the I/O access controller321aupdates at least one of the local cache area341aand mirror cache area342aof the RAM302awith cache data sent from the I/O access controller321bin the second CM300b.

Specifically, when the comparison at step S61has revealed that the first cache change count343ais smaller than the second cache change count344b, the I/O access controller321adiscards its cache data in the local cache area341aand then loads the emptied local cache area341awith cache data of the mirror cache area342bwhich is received from to the peer I/O access controller321b. Likewise, when the comparison at step S61has revealed that the second cache change count344ais smaller than the first cache change count343b, the I/O access controller321adiscards its cache data in the mirror cache area342aand then loads the emptied mirror cache area342awith cache data of the local cache area341bwhich is received from to the peer I/O access controller321b.

(Step S65) The I/O access controller321afurther updates at least one of the first and second cache change counts343aand344astored in the NVRAM309a. Specifically, when the comparison at step S61has revealed that the first cache change count343ais smaller than the second cache change count344b, the I/O access controller321aupdates the first cache change count343awith the value of the second cache change count344breceived from its peer I/O access controller321bin the second CM300b. Likewise, when the comparison at step S61has revealed that the second cache change count344ais smaller than the first cache change count343b, the I/O access controller321aupdates the second cache change count344awith the value of the first cache change count343breceived from its peer I/O access controller321bin the second CM300b.

Upon completion of the above initial setup process, the I/O access controller321aresumes access control operations in redundant operation mode.

According to the fourth embodiment described above, the I/O access controller in a booted CM determines what to do to start access control operations by consulting cache change counts, but not configuration data update counts. The procedure of the fourth embodiment is less complicated and thus more efficient than the third embodiment.

The foregoing third and fourth embodiments select a necessary procedure for starting access control operations, depending on cache change counts. By contrast, the fourth embodiment described below uses cache change flags, instead of cache change counts, to determine the same. Cache change flags indicate whether cache data has been changed in solitary operation mode.

FIG. 25illustrates how cache change flags are manipulated. Specifically, first and second cache change flags345aand346aare stored in the NVRAM309aof the first CM300a. The first cache change flag345aindicates whether data in the local cache area341ahas been changed during a period when the I/O access controller321ain the first CM300acontrols access in solitary operation mode. The second cache change flag346a, on the other hand, indicates whether data in the mirror cache area342ahas been changed during that same period. Both cache change flags345aand346aare given an initial value of zero.

Stored in the NVRAM309bof the second CM300bis another set of first and second cache change flags345band346b. The first cache change flag345bindicates whether data in the local cache area341bhas been changed during a period when the I/O access controller321bin the second CM300bcontrols access in solitary operation mode. The second cache change flag346bindicates whether data in the mirror cache area342bhas been changed during that same period. Both cache change flags345band346bare given an initial value of zero.

FIG. 26is a flowchart illustrating a process of setting cache change flags. While this example ofFIG. 26assumes that the I/O access controller321ain the first CM300aexecutes the illustrated process, its peer I/O access controller321bin the second CM300bcan also execute the process similarly.

(Step S81) In the initial state inFIG. 26, the first and second cache change flags345aand346ain the first CM300aare both zero. The I/O access controller321aproceeds to step S82when it starts access control operations in solitary operation mode (Yes at step S81). More specifically, the I/O access controller321astarts access control operations in solitary operation mode when it detects that the RoC device301bin the second CM300bhas stopped (e.g., T22inFIGS. 13and T42inFIG. 21).

(Step S82) The I/O access controller321adetermines, at regular intervals, whether any change has been made to the local cache area341aby, for example, a write request to volume Vol#0. If it is determined that there has been a change to the local cache area341a(Yes at step S82), the I/O access controller321aproceeds to step S84. Otherwise (No at step S82), the I/O access controller321aproceeds to step S83.

(Step S83) The I/O access controller321aalso determines, at regular intervals, whether any change has been made to the mirror cache area342aby, for example, a write request to volume Vol#1. If it is determined that there has been a change in the mirror cache area342a(Yes at step S83), the I/O access controller321aproceeds to step S87. Otherwise (No at step S83), the I/O access controller321areturns to step S82.

(Step S84) Now that a change to the local cache area341ahas been found at step S82, the I/O access controller321aalters the first cache change flag345ain NVRAM309afrom zero to one.

(Step S85) The I/O access controller321awatches whether any change has been made to the mirror cache area342aby, for example, a write request to volume Vol#1. If there has been a change to the mirror cache area342a(Yes at step S85), the I/O access controller321aproceeds to step S86.

(Step S86) The I/O access controller321aalters the second cache change flag346ain NVRAM309afrom zero to one.

(Step S87) Now that a change to the mirror cache area342ahas been found at step S83, the I/O access controller321aalters the second cache change flag346ain NVRAM309afrom zero to one.

(Step S88) The I/O access controller321awatches whether any change has been made to the local cache area341aby, for example, a write request to volume Vol#0. If there has been a change to the local cache area341a(Yes at step S88), the I/O access controller321aproceeds to step S89.

(Step S89) The I/O access controller321aalters the first cache change flag345ain NVRAM309afrom zero to one.

As can be seen from the above steps ofFIG. 26, the first cache change flag345ais set to one at the first change to the local cache area341aafter startup of the I/O access controller321ain solitary operation mode. Similarly, the second cache change flag346ais set to one at the first change to the mirror cache area342aafter startup of the I/O access controller321ain solitary operation mode.

FIG. 27is a flowchart illustrating what is executed when an RoC device starts up according to the fifth embodiment. The present embodiment executes, when an

RoC device boots up, a process similar to that ofFIG. 24, except that steps S61, S62, and S63are respectively replaced with steps S101, S102, and S103as seen inFIG. 27. The following description ofFIG. 27will focus on those modified steps.

(Step S101) When the I/O access controller321ain the first CM300acan reach its peer RoC device301bin the second CM300b(Yes at step S32), the I/O access controller321atests the values of cache change flags stored in the first CM300aand those stored in the second CM300b.

Specifically, the I/O access controller321areads first and second cache change flags345aand346aout of the NVRAM309avia the boot controller324a. The I/O access controller321aalso interacts with its peer I/O access controller321bin the second CM300bto read first and second cache change flags345band346bout of the NVRAM309bin the second CM300b. The I/O access controller321aproceeds to step S102if both cache change flags345band346bof the second CM300bare zero, and if at least one of the first and second cache change flags345aand346aof the first CM300ais one. The I/O access controller321aproceeds to step S104if both cache change flags345aand346aof the first CM300aare zero, and if at least one of the first and second cache change flags345band346bof the second CM300bis one. The I/O access controller321aproceeds to step S34if all the four cache change flags345a,345b,346a, and346bare zero. At step S34, the I/O access controller321aexecutes an initial setup process for access control, without duplexing cache data.

(Step S102) The I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. During the course of this initial setup process, the I/O access controller321aupdates cache data in the local cache area341band mirror cache area342bof the second CM300bwith its own cache data stored in the mirror cache area342aand local cache area341a, respectively.

Specifically, when the test at step S101has revealed that the first cache change flag345ais one, the I/O access controller321asends data from the local cache area341ato the I/O access controller321bin the second CM300band requests the I/O access controller321bto update the mirror cache area342bin the second CM300b. In response, the I/O access controller321bin the second CM300bdiscards its cache data in the mirror cache area342band then stores the received cache data in the emptied mirror cache area342b.

When the test at step S101has revealed that the second cache change flag346ais one, the I/O access controller321asends data from the mirror cache area342ato the I/O access controller321bin the second CM300band requests the I/O access controller321bto update the local cache area341bin the second CM300b. In response, the I/O access controller321bdiscards its cache data in the local cache area341band then stores the received cache data in the emptied local cache area341b.

(Step S103) During the course of the above initial setup process, the I/O access controller321aclears the cache change flag(s) of the first CM300ato zero if it or they were one at step S101. Upon completion of the above initial setup process, the I/O access controller321aresumes access control operations in redundant operation mode.

(Step S104) The I/O access controller321aexecutes an initial setup process in preparation for access control in redundant operation mode. During the course of this initial setup process, the I/O access controller321aupdates at least one of the local cache area341aand mirror cache area342aof the RAM302awith cache data sent from the I/O access controller321bin the second CM300b.

Specifically, when the test at step S101has revealed that the second cache change flag346bis one, the I/O access controller321adiscards its cache data in the local cache area341aand then loads the emptied local cache area341awith cache data of the mirror cache area342bwhich is received from to the peer I/O access controller321b. When the test at step S101has revealed that the first cache change flag345bis one, the I/O access controller321adiscards its cache data in the mirror cache area342aand then loads the emptied mirror cache area342awith cache data of the local cache area341bwhich is received from to the peer I/O access controller321b. Upon completion of the above initial setup process, the I/O access controller321aresumes access control operations in redundant operation mode.

As can be seen from the above description, the fifth embodiment relies on the cache change flags when the I/O access controller in a booted CM determines what to do to start access control operations. Each cache change flag only consumes a one-bit memory space to achieve the purpose of determining a procedure necessary for starting access control operations. Thus the fifth embodiment is more space-efficient than the fourth embodiment. Also it is obvious that the cache change flags are set or cleared less frequently than the cache change counts are incremented in the fourth embodiment. This is advantageous in terms of the total processing efficiency of CMs because changing cache change flags would impose little impact on the control operations even if those flags are located in a non-volatile memory device with a relatively slow access speed such as NVRAM.

As discussed in the second embodiment, the RoC device301ain the first CM300amay stop for some reason when its peer RoC device301bin the second CM300bis in the process of rebooting after abort in redundant operation mode. In such a situation, the foregoing second embodiment initiates a dual CM restoration process of two RoC devices301aand301bby forcibly stopping the RoC device301b. By contrast, the sixth embodiment described below is to cause the RoC device301bin the second CM300bto make the stopped RoC device301areboot in restoration mode, instead of initiating a dual CM restoration process.

FIGS. 28 and 29give a timing diagram illustrating how the sixth embodiment works when two RoC devices stop successively with a certain interval.

Specifically, this example assumes that the RoC device301bin the second CM300bstops first, and then the RoC device301ain the first CM300afollows.

Referring to the initial state seen inFIG. 28, the RoC device301ain the first CM300ais working together with the RoC device301bin the second CM300bin redundant operation mode. The latter RoC device301bthen stops working due to some anomalies (at time T61), and this event is detected by the I/O access controller321ain the first CM300a(at time T62). Upon detection, the I/O access controller321amakes the RoC device301bin the second CM300breboot in restoration mode. The I/O access controller321aalso causes the second CM300bto change its boot flag332bin the NVRAM309bfrom zero to one, as well as incrementing its configuration data update count331astored in the NVRAM309aof the first CM300a.

In response to the above request from the I/O access controller321ain the first CM300a, the RoC device301bin the second CM300bstarts rebooting in restoration mode. During this reboot process, the RAM302bin the second CM300bis allowed to retain the current data in its local cache area341band mirror cache area342b.

In the first CM300a, on the other hand, the I/O access controller321amakes its way to solitary operation mode, in which the I/O access controller321auses the local cache area341aand mirror cache area342ato continue access control operations for both volumes Vol#0and Vol#1. The I/O access controller321anow receives and executes a write request from a host device120to volume Vol#0or volume Vol#1and changes its cache data in the RAM302aaccordingly (at time T63).

Suppose here that the RoC device301ain the first CM300astops due to some anomalies (at time T64) before the RoC device301bin the second CM300bfinishes its reboot. When the RoC device301bin the second CM300breboots completely, the I/O access controller321bchecks its connection with the RoC device301ain the first CM300a(at time T65). Since the RoC device301ais stopped, the I/O access controller321bfinds it not possible to reach the RoC device301a.

Since the peer RoC device301acannot be reached, the I/O access controller321bchooses and executes an appropriate process depending on the boot flag332b, which is stored in the NVRAM309bof the second CM300b. Specifically, the I/O access controller321bcauses the RoC device301ain the first CM300ato reboot in restoration mode (at time T66) when the boot flag332bis one. The I/O access controller321bthen waits for its peer I/O access controller321ain the first CM300ato become reachable. When The RoC device301ain the first CM300ais completely rebooted as requested by its peer the RoC device301bin the second CM300b, the two I/O access controllers321aand321bsuccessfully confirm their connection over communication path P1(at time T67). Here the I/O access controllers321aand321bcompare the configuration data update count331ain NVRAM309aof the first CM300awith the configuration data update count331bin NVRAM309bof the second CM300b.

Because the configuration data update count331aof the first CM300ais greater than the configuration data update count331bof the second CM300b, the I/O access controllers321aand321bconfirm the validity (up-to-dateness) of cache data stored in the first CM300aand execute an initial setup process in preparation for access control in redundant operation mode (at time T68). The configuration data update count331bin the second CM300bis incremented at the start of this initial setup process, which renders the two configuration data update counts331aand331bin a synchronized state.

During the course of the initial setup process started at time T68, the I/O access controllers321aand321bexecute a task of duplexing cache data held in the first CM300a. More specifically, the I/O access controller321asends data from its own local cache area341aand mirror cache area342ain the RAM302ato the peer I/O access controller321bin the second CM300b. The I/O access controller321bdiscards data in its local cache area341band mirror cache area342bof the RAM302b, and loads the emptied mirror cache area342bwith new data sent from the local cache area341a, as well as the emptied local cache area341bwith new data sent from the mirror cache area342a. The I/O access controllers321aand321bstart, upon completion of their respective initial setup processes, access control operations in redundant operation mode (at time T69).

As can be seen from the above operation, the I/O access controller321bin the second CM300bcannot establish a connection with its peer RoC device301ain the first CM300aat time T65, and finds that its own boot flag332bhas been set to one. In this case, the I/O access controller321bmakes the RoC device301ain the first CM300areboot in restoration mode, instead of initiating a dual CM restoration process. This processing enables the I/O access controllers321aand321bto resume access control operations in a shorter time than in the second embodiment.

FIG. 30is a flowchart illustrating what is executed when an RoC device starts up according to the sixth embodiment. The present embodiment executes, when an RoC device boots up, a process similar to that ofFIGS. 17 and 18, except that step S41is replaced with steps S111and S112as seen inFIG. 30. The following description ofFIG. 24will focus on those modified steps.

(Step S111) When the rebooted RoC device301bin the second CM300bis not reachable (Yes at step S32), and when the boot flag332aof the first CM300ahas been set to one (step S39), the I/O access controller321afirst clears the boot flag332ato zero (step S40) and then requests, through the boot controller324a, the RoC device301bin the second CM300bto reboot in restoration mode. Steps S40and S111may, however, be executed in the opposite order.

(Step S112) The I/O access controller321awatches for a predetermined time after step S111whether the rebooted RoC device301bin the second CM300bcan be reached. The duration of this watching may be equal to or somewhat longer than the time that the RoC device301brequires to become reachable from the peer RoC device301aafter it is rebooted in restoration mode.

The I/O access controller321aproceeds to step S33ofFIG. 17to continue its operation from that point, if the RoC device301bin the second CM300bhas become reachable within the predetermined period since the start of step S111. The series of steps S40, S111, S112, and S33may correspond to, for example, the foregoing process after time T65ofFIG. 28.

The I/O access controller321a, on the other hand, proceeds to step S42if the RoC device301bin the second CM300bfails to recover its connectivity (i.e., if the RoC device301bdoes not reboot properly) in the predetermined period since the start of step S111. This path from step S112to step S42may correspond to, for example, the foregoing process executed in the case where the I/O access controller321bin the second CM300bcannot reach the RoC device301ain the first CM300aat time T67inFIG. 29. In this case, the I/O access controller321bstarts access control operations for both volumes Vol#0and Vol#1in solitary operation mode.

The above-described processing ofFIG. 30is a variation of the second embodiment. More particularly, it is another version of step S41ofFIG. 18. This variation is, however, not limited to the second embodiment, but may also be applied to any of the foregoing third, fourth, and fifth embodiments. More specifically, the I/O access controller321amay proceed from step S112ofFIG. 30to step S33ofFIG. 23, or step S61ofFIG. 24, or step S101ofFIG. 27, instead of going to step S33ofFIG. 17, when the RoC device301bin the second CM300bbecomes reachable in the predetermined period.

Various embodiments of the proposed storage system, control apparatus, and control method have been described above. The described techniques prevent redundancy-protected control devices in a storage system from losing their cache data even when both of them are stopped successively.