Fault-tolerant computer systems

Abstract: A computer system comprises a number of replicated units. At least two of the units include monitoring components for monitoring the status of the system. The monitoring components communicate with each other to exchange status information, so as to enable both of the monitoring components to form a combined view of the status of the system, based on information gathered by both of the monitoring components. This enables each monitoring component to locate faults more precisely, and thus enables more appropriate recovery action to be taken.

Description: Background to the invention This invention relates to fault-tolerant computer systems. It is well known that fault tolerance can be achieved in a computer system by replicating components of the system. For example, two processing units may be provided, one of which acts as the active unit and the other of which acts as a stand-by. When a fault is detected in the active unit, from which it cannot recover, the stand-by is switched in as the active unit, allowing system operation to continue. FR-A-2 561 410 describes a such a system, comprising two processing units. Each processing unit includes an image memory, which stores the values of the inputs and outputs of that unit during each program cycle. The two units exchange these values, by way of a serial connection, and each compares the values stored locally with those obtained from the other processing unit. If there is a mismatch, a fault procedure is activated, which attempts to determine the nature of the fault. A problem which arises in such a system is how to locate faults precisely, to enable an appropriate recovery action to be taken. In particular, in the above-referenced French patent, the fact that a mismatch is detected merely indicates that a fault is present somewhere in the system; it does not indicate where that fault is located. Summary of the invention The present invention provides a computer system as claimed in Claim 1 below. It can be seen that in the present invention, rather than merely exchanging input and output values, the two processing units exchange status reports relating to the various system units. For example, a status report might indicate that a data storage unit has disappeared as viewed by one monitoring means, or else might indicate that the data storage unit is operating normally. From these status reports, it is possible for each processor to deduce a combined view of the status of each system unit, from which it is possible to locate faults more precisely than would be possible merely on the basis of comparing input and output values. Brief description of the drawings Figure 1 is an overall block diagram of a fault-tolerant computer system embodying the invention. Figure 2 is a block diagram of a high availability manager (HAM) component of the system. Figure 3 is a flow chart showing the operarion of the HAM. Description of an embodiment of the invention One computer system in accordance with the invention will now be described by way of example with reference to the accompanying drawings. Referring to Figure 1, the system comprises two processing units 10 and 11. Each of these processing units may comprise, for example, an ICL DRS 6000 processor, running under the UNIX operating system. The processing units both have access to two data storage units 12 and 13. Each data storage unit may, for example, comprise a cabinet holding a number of disk drives. The processing units are interconnected by one or more local area networks (LANs) 14. In operation, one of the processing units 10, 11 acts as the active unit, and the other is the stand-by. If a fault condition is detected which indicates a failure of the active processing unit, the system can be switched over to make the other processing unit the active unit, allowing the operation of the system to continue. The data storage units 12, 13 are used in parallel, with data being mirrored between the two units. Hence, if one of the data storage units fails, the data is still available and system operation can continue. A further data storage unit (not shown) may also be provided as a stand-by so that mirror operation may be continued after failure of one of the units. Each of the processing units 10, 11 runs a software component referred to herein as the high availability manager (HAM), the purpose of which is to provide a system monitoring and management facility. Referring to Figure 2, this shows one of the HAMs. The HAM 20 comprises a number of sub-test modules 21 which perform various checks on the local processor (ie the processor in which the HAM is running) and on the units connected to it. For example, these checks include checking the interfaces between the processor and the data storage units. The test modules 21 generate a sequence of status reports. As will be described, the HAM also receives status reports from the HAM in the remote processing unit. The status reports, from the test modules and from the remote HAM, are fed by way of a UNIX pipe 22 to a monitoring component 23. The monitoring component runs periodically, every two to three seconds, to read the status reports from the pipe 22 and to take the appropriate actions. The monitoring component 23 maintains a system status table 24, which holds an entry for each of the sub-tests, containing the latest status report from that sub-test. Each entry includes an age value which is reset to a predetermined initial value whenever a status report is received from the corresponding sub-test. All the age values in the status table are periodically decremented by the monitoring component. Hence, each age value indicates how long the corresponding status report has been valid. The monitoring component also has access to a configuration file 25, containing a number of records. Each record specifies a particular combination of one or more status reports, and specifies an action to be performed when this combination occurs. The action may, for example comprise displaying an error message to the operator, or performing some automatic fault recovery action. Each record in the configuration file may also optionally contain a time value, which specifies how long the status report must be valid before the action is triggered. This is to allow time for the other processor to forward its report about a particular fault situation, so that a system-wide decision can be made about the location of the fault and what action to take. A status report is considered to be "mature" if its age value in the status table is less than this time value. As will be described in more detail below, the monitoring component scans the configuration file to find a record that matches the currently reported fault conditions in the status report file. If a record is found, the action specified in that record is performed. Referring now to Figure 3, this is a flow chart showing the operation of the monitoring component 23. (301) The monitoring component gets a status report from the pipe 22. (302) The monitoring component then checks the status report to see whether it is good or bad, ie whether it indicates that new fault condition has been detected. (303) If the status report is bad, a check is made to see whether the status report came from the local processing unit, or from the remote processing unit. (304) If the status report came from the local processing unit, a message is generated, containing the status report, and is sent to the remote processing unit over the LAN 14. When the HAM in the remote processing unit receives this message, it passes the status report to its pipe 22. (305) The monitoring component then updates the appropriate entry in its local status table 24, by writing the status report into the entry and resetting the age value of the entry. (306) If the status report was good, the monitoring component then returns to step 301 above, to get the next status report from the pipe 22. If, on the other hand, the status report was bad, the following steps are performed. (307) The configuration file 25 is opened. (308) A record is read from the configuration file. (309) The status table 24 is then examined, to see whether the particular report or combination of reports specified by the record is present. (310) If the specified combination of status reports is present, these reports are checked to see whether they are mature, ie whether they have been valid for a length of time greater than the time value (if any) specified in the configuration file record. This involves comparing the age values in the status table entries with the time values in the configuration file record. (311) If the reports are mature, the monitoring component generates an error message, which indicates the actions specified in the configuration file record. The associated entries in the status table 24 are marked as "reported". Once marked in this way, an error condition is ignored by the monitoring component, until the condition changes or is cleared. (312) The configuration file is then closed, and the monitoring component returns to step 301 above, to get the next status report. (313) If the specified combination of status reports is not present, or if the reports are not yet mature, a check is made to see whether the end of the configuration file has been reached. If not, the monitoring component returns to step 308 above, to read the next record from the configuration file. (314) If the end of the configuration file is reached, a check is made to see whether any records in the configuration file have been found relating to the current status report. (315) If no valid records were found, an error message is generated, indicating that an unrecognised fault condition has been detected. The monitoring component then proceeds to step 312, so as to close the configuration file and to get the next status report. In summary, it can be seen that each HAM makes decisions about the location of faults on the basis of the contents of its status report file. Each status report file holds status reports generated by both HAMs. Hence, each HAM is able to analyze a fault on the basis not only of local information, gathered by the local test modules 21, but also information sent to it from the HAM in the remote processor. Thus, each HAM is able to locate a fault on the basis of a global view of the system, rather than merely a local view. For example, if one of the processing units has a failure on one of its data storage interfaces, its HAM will detect that the corresponding data storage unit has disappeared, and will send a status report message to the HAM in the other processing unit to inform it of this. Meanwhile, the HAM in the other processing unit will detect that both data storage units are working normally, and will send a status report message to the HAM in the first processing unit to inform it of this. Thus, both HAMs will be able to deduce from the status reports in their status tables that a data storage interface on the first processing unit has failed, and both will take suitable action to recover from this fault, eg by switching over to make the second processing unit the active processor. It will be appreciated that without the exchange of information between the HAMs, neither HAM would have been able to identify the location of this fault.

Claims: A computer system comprising: (a) first and second processing units (10, 11); and (b) a plurality of further system units (12, 13), connected to both said first and second processing units; (c) each of the processing units (10,11) including a monitoring means (20); (d) the processing units (10, 11) being inter-connected by way of a data connection (14); characterised in that : (e) each of the monitoring means (20) performs checks on its local processing unit (10 or 11) and on the further system units (12, 13) connected to that unit, and generates status reports indicating the status of those units as viewed by that monitoring means; (f) the monitoring means (20) exchange their status reports, by way of said connection (14); and (g) each of the monitoring means (20) deduces from the status reports generated locally, and from those received from the other monitoring means, a combined view of the status of the units. A system according to Claim 1 wherein each of said monitoring means (20) includes a status table (24) for holding a plurality of status reports, including both the locally generated status reports and those received from the other monitoring means (20). A system according to Claim 2 wherein each of said monitoring means (20) includes means for searching its status table (24) for a predetermined combination of status reports, and for initiating a predetermined action upon detection of that combination. A system according to Claim 3 wherein said predetermined action is initiated only if the status reports in said predetermined combination have been valid for at least a predetermined period of time.

Inventors: TAYLOR RICHARD NORCOTT, TAYLOR, RICHARD NORCOTT
Applicants: INT COMPUTERS LTD, INTERNATIONAL COMPUTERS LIMITED
Application year: 1994
Main classification: G06F11
IPC codes: G06F11, G06F15, G11C29