Method of recovering failure of online control program

Abstract: A processing sytem comprises a plurality of CPUs, a com­ mon storage device which are shared by all the CPUs, online control programs which are executed by the respective CPUs, and monitor programs which monitor the states of the online control programs and which control the online control programs so that, when the failure of the online control program has occurred, the process of the online control program having failed can be taken over by the normal online control program. Recovering the failure of an online control program is done by storing process taking-over information in said common storage device separately from log.

Description: Background of the Invention Field of the Invention The present invention relates, in a processing system having a plurality of CPUs each executing an online control program, to a method of recovering the failure of the online control program. Description of the Prior Art A prior-art method of recovering the failure of an online control program at high speed is called the "hot standby recovery method." With this method, in correspondence with the online control program, a standby program is previously prepared in the state in which the process of the preceding stage of a recovery process for the failure has been completed, and the recovery process is performed with the standby program at the failure of the online control program, thereby to shorten the recovery period of time. The official gazette of Japanese Patent Application Laid-open No. 11457/1984 is relevant to the technique of this type. The hot standby recovery method in the prior art has had such problems that the redundant construction in which the standby program is prepared separately from the online control program is employed, that at the failure of the online control program, a service cannot be started until the recovery process with the standby program is completed, and that the online control program cannot be recovered at high speed in a case where it has failed after the failure of the standby program. A further method for recovering from a failure of a computer system is disclosed in EP-A-221 274. According to this method, an active processor, i.e. a processor for normal user-transaction processing, has an associated backup processor. When the active processor fails, the backup processor performs a recovery processing and takes over user-transaction processing as new active processor. As long as no failure occurs in the system, the backup processor is mostly idle and from time to time monitors the normally active processor. This system therefore suffers from the same disadvantage as the above-mentioned prior art in that redundancy is provided which makes the system inefficient. Summary of the Invention An object of the present invention is to provide a method for recovering from a failure of an online control program at high speed, which utilizes a computer system efficiently. The present invention for accomplishing such an object is characterized in that a common storage device which is shared by a plurality of processors stores process taking-over information separately from log, and that, when the online control program of a certain processor has failed, another processor inhibits the use of resources being processed by the failed online control program in view of the process taking-over information, fetches an unprocessed service and then starts it by the use of an available resource and executes a recovery process by the use of the log. Brief Description of the Drawings Fig. 1 is a diagram showing the system architecture of an embodiment of the present invention; Fig. 2 is a flow chart of the processing of a monitor program corresponding to an online control program which takes over a service at the failure of a certain online control program; Fig. 3 is a flow chart of the failure processing of the online control program which takes over the service; Fig. 4 is a flow chart of the restart processing of the monitor program; Fig. 5 is a flow chart of the processing of another monitor program after the restart processing; and Fig. 6 is a flow chart of the processing of the online control program after being restarted. Detailed Description of the Preferred Embodiments Fig. 1 is a diagram showing the system architecture of an embodiment of the present invention. Referring to the figure, numerals 11, 12 and 13 designate CPUs, which are loosely coupled by communication paths 191, 192 and 193. In the respective CPUs, there are online control programs 112, 122 and 132 and monitor programs 111, 121 and 131 therefor. Numerals 162, 172 and 182 indicate terminals which receive the services of the online control programs. The terminals 162 receive the services of the online control program 112 through a communication controller 16. Likewise, the terminals 172 receive the services of the online control program 122 through a communication controller 17, and the terminals 182 receive the services of the online control program 132 through a communication controller 18. Shown at numeral 14 is a common storage device, which is shared by the respective CPUs and is connected with paths 113, 123 and 133. In the common storage device 14, there are items of taking-over information 141, 142 and 143 which correspond to the respective online control programs 112, 122 and 132. Numeral 151 indicates a file for databases which the online control programs use, and numerals 152, 153 and 154 indicate files having log items which correspond to the online control programs 112, 122 and 132, respectively. Service request queues are also stored in the files 152, 153 and 154. The file for the databases, 151, the files for the log items etc., 152, 153 and 154, and the communication controllers 16, 17 and 18 are physically connected to the respective CPUs by paths 114, 124, 134, 161, 171 and 181, and they are accessible from all the CPUs. The taking-over information items 141, 142 and 143 include information items which indicate the names of resources (files) being processed and being available at present, and the pointer information items of the service request queues. Now, processing in the case where the online control program in the CPU has failed in the architecture of Fig. 1 will be described with reference to flow charts in Figs. 2 - 6. In the flow charts, Fig. 2 shows the processing of the monitor program corresponding to the online control program which takes over a service at the failure of the certain online control program, Fig. 3 shows the failure processing of the online control program which takes over the service, Fig. 4 shows the restart processing of the monitor program, Fig. 5 shows the processing of the other monitor program after the restart processing, and Fig. 6 shows the processing of the online control program after being restarted. When the online control program 112 in the CPU 11 has failed, the monitor program 111 senses the failure, and it communicates the failure to the respective monitor programs 121, 131 in the CPUs 12, 13 through the communication paths 191, 192. Which of the online control programs 122 and 132 takes over the service of the online control program 112 is determined as priority levels beforehand, and the online control program 122 does in conformity with the priority levels. Upon receiving the communication from the monitor program 111, the monitor program 121 reads out the taking-over information 141 of the online control program 112 from the common storage device 14 (Step 21 in Fig. 2), and it transfers the taking-over information to the online control program 122 and informs this program of a processing start (Step 22 in Fig. 2). The online control program 122 which has started processing performs the following three steps in parallel: [1] On the basis of the name of resources being processed as included in the taking-over information, the use of the resources which need to be recovered is inhibited (Step 31 in Fig. 3). [2] The communication paths 161 are changed-over to recover the states of the terminals 162 so that the terminals 162 having received the service of the online control program 112 through the communication controller 16 may be permitted to receive the service of the online control program 122 (Step 32 in Fig. 3). [3] A program for processing a message for the service is initiated (Step 33 in Fig. 3). Subsequently to the end of the processing [1], a process for recovering databases is executed (Step 35 in Fig. 3). After all the processing steps [1], [2] and [3] have ended (Step 34 in Fig. 3), the service for the terminals 162 is restarted by reference to the service request queue based on the pointer in the taking-over information thereof (Step 36 in Fig. 3), and the processing flow shifts to a normal process (Step 38 in Fig. 3). Meantime, services concerning the resource the use of which has been inhibited cannot be offered, but the other services can be rendered as usual. After the database recovering process has terminated (Step 35 in Fig.3), the notice of the termination is given (Step 37 in Fig. 3), and the inhibition of the use of the resource is released. Thenceforth, the normal service is performed in a degenerate form. There will be described processing in which the online control program having failed is restarted thereby to be restored into the normal state thereof. The online control program 112 is restarted by the CPU 11. The monitor program 111 starts the online control program 112 (Step 41 in Fig. 4), and after start processing (Step 43 in Fig. 4), the online control program 112 waits a notice from the monitor program 111 without rendering any service (Step 44 in Fig. 4). After the start processing of the online control program 112 has terminated, the monitor program 111 informs the monitor programs 121 and 131 of the termination (Step 42 in Fig. 4). When the monitor program 121 informed of the termination gives the online control program 122 the notice of the termination of the start processing of the online control program 112 (Step 51 in Fig. 5), the online control program 122 stops the service which it has been substitutionally executing (Step 53 in Fig. 5), and it writes taking-over information into the common storage device 14 (Step 54 in Fig. 5) and gives the monitor program 121 the notice of the termination of the substitution service (Step 55 in Fig. 5). The monitor program 121 informs the monitor program 111 of the termination (Step 52 in Fig. 5). When informed, the monitor program 111 returns to the processing in Fig. 2, in which it reads out the taking-over information 141 of the common main memory 14 (Step 21 in Fig. 2) and transfers the taking-over information to the online control program 112 waiting the notice of the processing start, thereby to inform this program 112 of the processing start (Step 22 in Fig. 2). The online control program 112 informed of the processing start executes the two steps in parallel; [1] on the basis of the name of a resource being processed as included in the taking-over information, the use of the resource which needs to be recovered is inhibited (Step 61 in Fig. 6), and [2] the communication paths are changed-over to recover the states of the terminals 162 so that the terminals 162 having received the service of the online control program 122 through the communicatior controller 16 may be permitted to receive the service of the online control program 112 (Step 62 in Fig. 6). The initiation of a message processing program for the service has been done by the start processing. After both the processing steps [1] and [2] have ended (Step 63 in Fig. 6), the service for the terminals 162 is restarted by reference to the service request queue based on the pointer in the taking-over information thereof (Step 64 in Fig. 6), and the processing flow shifts to a normal process (Step 65 in Fig. 6). Since this situation differs from the failure of the online control program, any recovering process need not be executed. In the above way, the system is restored into the original state. The embodiments of the present invention described above are summed up as follows: (1) The name of resources being processed and the head pointer of a service request queue are always put in a common main memory, for access by every online control program and as information for taking over processing at the occurrence of a failure. (2) When an online control program has failed, another online control program being normal inhibits the use of resources being processed by reference to the taking-over information in the common storage device and starts a recovering process by the use of a log. (3) In parallel with the recovering process, an unprocessed service is fetched from the service request queue on the basis of the service request queue pointer, and a service which employs usable resources is started. A service which employs the resource having had its use inhibited is set as an error. (4) When the failure recovering process terminates, the resource the use of which has been inhibited is made usable. (5) After the online control program having failed is restarted, the service which has been executed substitutionally by the other online control program is returned to the original online control program by processing similar to that of the failure recovery. According to the foregoing embodiments, the information in the common storage device need not be recovered even at the failure of the online control program, so that the recovering process and the service taking-over can be independently performed by placing the service taking-over information in the memory, and that the period of time till service restarting is shortened by executing both the recovering and taking-over processes in parallel. A standby program can be dispensed with by endowing the online control programs with the function of taking over the service request queues. When the common storage device has failed, it is subjected to the prior-art recovering process, whereby it can be recovered in a time interval equal to that of the prior art even in the worst case.

Claims: A method of recovering from a failure of an online control program in a processing system having a plurality of processors (11, 12, 13) each of which executes an online control program, a memory (14) which is common to the processors, and a plurality of resources (151) which are used by the online control program, the method comprising the steps of: (1) executing the online control program in each of said processors and providing service to a corresponding terminal, (2) storing by each processor taking-over information, which is used for take-over of a process of the respective online control program by another processor, and log information, which is used by a recovery process, into said memory (14), (3) inhibiting use of a resource, which was used when the failure occurred in the online control program of a first processor, by the online control program of a second processor and by reference to the taking-over information stored in said memory (14), (4) executing a recovery process by the online control program of said second processor on the basis of the log information in said memory (14), and (5) fetching an unprocessed service according to the taking-over information in said memory (14) and starting a service requiring a resource which is usable, by the online control program of said second processor in parallel with the executing step (4). A method according to claim 1, wherein said taking-over information includes a name of a resource being processed and a pointer to a service request queue. A method according to claim 1 or 2, further comprising the steps of: (6) restarting the online control program of said first processor, and (7) returning service by the online control program of said second processor to the online control program of said first processor.

Inventors: YAMAMOTO YOUICHI, YAMAMOTO, YOUICHI
Applicants: HITACHI LTD, HITACHI, LTD.
Application year: 1988
Main classification: G06F11
IPC codes: G06F11