Abstract:
A system includes a plurality of apparatuses connected to a network, wherein each apparatus monitors the occurrence of trouble in a monitoring object apparatus which is one of the apparatuses in the system. Further, each apparatus is monitored for the occurrence of trouble in the apparatus by a monitor apparatus which is another one of the apparatuses in the system. The system enables adding and subtracting apparatuses to the system, as well as restoring apparatuses which were previously subtracted due to detected troubles which have been corrected.

Description:
BACKGROUND OF THE INVENTION 
   1. Field of the Invention 
   The present invention relates to a monitoring technology for monitoring whether or not trouble has occurred in an apparatus connected to a network, such as a computer, a telephone exchange system or a process controller. Particularly, the present invention relates to a mutual monitoring technology in which an apparatus connected to a network monitors whether or not trouble has occurred in another apparatus connected to the network. 
   2. Description of the Related Art 
   Technologies in which an apparatus connected to a network monitors another apparatus connected to a network include the following technologies. 
   In the related art described in Japanese Patent Published Application No. 2001-75837, each computer to be monitored regularly sends to other computers a packet including its identifier. The packet is sent by broadcast or multicast. Each computer contains a component information storing portion where identifiers of all computers to be monitored are registered. If the identifier of a computer is registered in that storing portion of other computers and a packet from that computer has not been received within a predetermined time, the other computers determine that trouble has occurred in that computer from which a packet has not been received. 
   This related art has a problem of increasing the load on a network because it is necessary to send a packet by broadcast or multicast. Further, this related art has a problem of requiring a large memory size for the component information storing portion if there are a large number of computers to be monitored. It is also not easy to add a new computer to be monitored. This is because it is required that the identifier of all computers to be monitored should be registered in the storing portion of each computer for use in broadcast or multicast. 
   In the related art described in Japanese Patent Published Application No. 2002-312199, each computer to be monitored has a list showing an order of inquiry of each computer. Namely, in this list, identifiers of all computers to be monitored are linked by pointers in an order of inquiry. In this list, the pointer of the last computer shows the identifier of the first computer. That is, this list is a ring type order list. Each computer sends back a response upon inquiry from another computer. Each computer recognizes the first computer to which the inquiry is made based on that ring type order list retained by itself, and inquires of that first computer. If a response is given to that inquiry from the first computer, it is determined that the first computer is normal, and processing is ended. If no response is given from the first computer, it is determined that trouble has occurred in the first computer. In this case, further, each computer inquires of the second computer in the order of inquiry based on the ring type order list. If a response is given from the second computer, it is determined that the second computer is normal, and processing is ended. In this case, the second computer which sends back a response, performs processing same as the processing described above based on the ring type order list retained by the second computer. 
   This related art has a problem of requiring a large memory size when there area large number of computers to be monitored. It is also not easy to add a new computer to be monitored. This is because it is necessary for each computer to retain the ring type order list in which identifier of all computers to be monitored is linked by pointers. 
   SUMMARY OF THE INVENTION 
   According to the present invention, each mutual monitoring apparatus in the mutual monitoring system can monitor whether or not trouble has occurred in another mutual monitoring apparatus with a small memory size even if there are a large number of mutual monitoring apparatuses to be monitored in the mutual monitoring system. A new mutual monitoring apparatus can easily be added to the mutual monitoring system. 
   The reason for this is that each mutual monitoring apparatus retains only information about a monitoring object apparatus and information about a monitor apparatus. Namely, since only information about two apparatuses is retained, the memory size can be reduced as compared to the related art in which identifiers of all mutual monitoring apparatuses to be monitored should be retained. The number of other monitoring apparatuses for which information should be changed when a new monitoring apparatus to be monitored is added to the mutual monitoring system can be also reduced. 
   The favorable effect described above of the present invention is achieved as explained below. 
   In the present invention, a mutual monitoring system comprises a plurality of mutual monitoring apparatuses connected to a network. The mutual monitoring apparatus in the mutual monitoring system monitors the occurrence of trouble in a monitoring object apparatus which is one of the mutual monitoring apparatuses in the mutual monitoring system and, is also monitored for the occurrence of a trouble in that mutual monitoring apparatus by a monitor apparatus which is another one of the mutual monitoring apparatuses in that mutual monitoring system. 
   The mutual monitoring apparatus, in an exemplary embodiment, comprises a management information storing portion in which a monitoring object identifier and a monitor identifier are registered, a monitoring unit that monitors the monitoring object apparatus indicated by the monitoring object identifier, and a responding unit that responds to the monitoring of the monitor apparatus indicated by the monitor identifier. 
   The mutual monitoring system, in the exemplary embodiment, further comprises separation processing unit. That unit, for exemplary purposes, may comprise a searching unit and a management information changing unit. The searching unit searches for a search target apparatus in the mutual monitoring apparatuses in the mutual monitoring system. The management information changing unit changes the monitoring object identifier registered in the mutual monitoring apparatus to the identifier of the search target apparatus, and changes the monitor identifier registered in the search target apparatus to the identifier of the mutual monitoring apparatus. 
   The searching unit, in an exemplary embodiment, searches the search target apparatus by sequentially requesting other mutual monitoring apparatuses in the mutual monitoring system to send back the monitor identifier registered in the requested mutual monitoring apparatus, and determines as the search target apparatus the mutual monitoring apparatus which sends back the monitor identifier which indicates the monitoring object apparatus of the mutual monitoring apparatus. Such request, for example, begins with the monitor apparatus of the mutual monitoring apparatus and then is sequentially made to the monitor apparatus of the mutual monitoring apparatus sending back the monitor identifier in response to the request. 
   The searching unit also, in the exemplary embodiment, searches the search target apparatus by requesting by broadcast other mutual monitoring apparatuses in the mutual monitoring system to send back the monitor identifier registered in the requested mutual monitoring apparatus, and determines as the search target apparatus the mutual monitoring apparatus which sends back the monitor identifier which indicates the monitoring object apparatus of the mutual monitoring apparatus. 
   In the present invention, in the exemplary embodiment, 1) the mutual monitoring apparatus further comprises a log information storing portion in which log information of the monitoring object apparatus are registered, and a log collecting unit that collects log information of the mutual monitoring apparatus, 2) the responding unit sends back the log information of the mutual monitoring apparatus collected by the log collecting unit to the monitor apparatus in response to a log acquirement request from the monitor apparatus, 3) the monitoring unit sends the log acquirement requests to the monitoring object apparatus and registers the log information sent back in the log information storing portion, and if the log information is not sent back, determines that trouble has occurred in the monitoring object apparatus, and 4) the separation processing unit of the mutual monitoring apparatus further comprises a log information updating unit that updates the log information registered in the log information storing portion with the log information received from the monitoring object apparatus. 
   The mutual monitoring apparatus, in the exemplary embodiment, may further comprise a selection unit that selects a first mutual monitoring apparatus and a second mutual monitoring apparatus from the mutual monitoring apparatuses in the mutual monitoring system and a management information managing unit that registers the identifier of the mutual monitoring apparatus as the monitor identifier of the first mutual monitoring apparatus and as the monitoring object identifier of the second mutual monitoring apparatus, registers the identifier of the first mutual monitoring apparatus as the monitoring object identifier of the mutual monitoring apparatus, and registers the identifier of the second mutual monitoring apparatus as the monitor identifier of the mutual monitoring apparatus. 
   The invention includes a mutual monitoring method and a signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital programming apparatus. The method and program are implemented in a mutual monitoring system comprising a plurality of the mutual monitoring apparatuses connected to a network, and with a management information storing portion in which a monitoring object identifier and a monitor identifier are registered. The program executes the respective steps of the mutual monitoring method according to the present invention. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is a block diagram showing an example of the configuration of a mutual monitoring system according to the present invention; 
       FIG. 2  is a block diagram showing an example of the configuration of a mutual monitoring apparatus  1 A; 
       FIG. 3  is a block diagram showing an example of the configuration of separation processing means  15 A; 
       FIG. 4  is a block diagram showing an example of the addition processing means  16 A; 
       FIGS. 5A and 5B  are views for explaining an operation when trouble occurs in a mutual monitoring apparatus  1 B; 
       FIG. 6  is a flowchart showing an example of processing by monitoring means  12 A; 
       FIG. 7  is a flowchart showing an example of processing by responding means  13 B; 
       FIG. 8  is a flowchart showing an example of processing by searching means  151 A when instructed to start separation processing; 
       FIG. 9  is a flowchart showing an example of processing by searching means  151 D when receiving an inquiry about an address of a monitor; 
       FIG. 10  is a flowchart showing an example of processing by log information changing means  153 A when instructed to start processing; 
       FIG. 11  is a flowchart showing an example of processing by log information changing means  153 C when receiving a first log information request; 
       FIG. 12  is a flowchart showing an example of processing by management information changing means  152 A when instructed to start processing; 
       FIG. 13  is a flowchart showing an example of processing by management information changing means  152 C when receiving a first address change request; 
       FIGS. 14A and 14B  are views for explaining an operation when adding a mutual monitoring apparatus  1 E to the mutual monitoring system; 
       FIG. 15  is a flowchart showing an example of processing by selection means  161 E when instructed to start addition processing; 
       FIG. 16  is a flowchart showing an example of processing by selection means  161 A,  161 B,  161 C and  161 D when receiving an IP address inquiry; 
       FIG. 17  is a flowchart showing an example of processing by log information managing means  163 E when instructed to start processing; 
       FIG. 18  is a flowchart showing an example of processing by log information managing means  163 B when receiving a second log information request; 
       FIG. 19  is a flowchart showing an example of processing by management information managing means  162 E when instructed to start processing; 
       FIG. 20  is a flowchart showing an example of processing by management information managing means  162 B when receiving a second address change request; 
       FIG. 21  is a flowchart showing an example of processing by management information managing means  162 A when receiving a third address change request; and 
       FIG. 22  is a flowchart showing an example of processing by searching means  151 A when searching a new monitoring object apparatus by broadcast. 
   

   DETAILED DESCRIPTION OF THE INVENTION 
   An embodiment of the present invention will be described in detail below with reference to the drawings. In the explanations described below, suffixes B, C, D, . . . are added when means and the like in the apparatuses  1 B,  1 C,  1 D, . . . are shown. For example, when monitoring means belonging to the apparatus  1 C is shown, it is described as monitoring means  12 C. 
     FIG. 1  is a block diagram showing an example of the configuration of an embodiment of a mutual monitoring system according to the present invention. In the example of the configuration, the mutual monitoring system is comprised of a plurality of mutual monitoring apparatuses  1 A to  1 D, each of which is an apparatus such as computers connected to a network, a management server  2 , and a network  3  for connecting those apparatuses. In this embodiment, mutual monitoring apparatuses  1 A,  1 B,  1 C and  1 D monitor mutual monitoring apparatuses(from hereon in the detailed description of the invention simply referred as apparatus)  1 B,  1 C,  1 D and  1 A, respectively. 
     FIG. 2  is a block diagram showing an example of the configuration of an apparatus  1 A. In the example of the configuration, the apparatus  1 A comprises sending/receiving means  11 A, monitoring means  12 A, responding means  13 A, configuration changing means  14 A, log collecting means  17 A, a storage apparatus  18 A, an input portion  21 A such as a keyboard, and a display portion  22 A such as a liquid crystal display (LCD). 
   The storage apparatus  18 A is implemented with magnetic disks, optical disks or the like, and has a log information storing portion  19 A and a management information storing portion  20 A. 
   The log information storing portion  19 A comprises an own log storing portion  191 A in which its own log information of the apparatus  1 A is registered, and a monitoring object log storing portion  192 A in which log information of the monitoring object apparatus of the apparatus  1 A is registered. The monitoring object apparatus of the apparatus  1 A is one of the apparatuses in the mutual monitoring system to be monitored for the occurrence of a trouble in that apparatus by the apparatus  1 A. The management information storing portion  20 A comprises a monitoring object address storing portion  201 A in which an identifier, for example an IP address, of the monitoring object apparatus of the apparatus  1 A is registered, and a monitor address storing portion  202 A in which an identifier, for example an IP address, of the monitor apparatus is registered. The monitor apparatus of the apparatus  1 A is one of the apparatuses in the mutual monitoring system to monitor occurrence of a trouble in the apparatus  1 A. Under normal circumstances or conditions the monitoring object apparatus and the monitor apparatus of the apparatus  1 A are different apparatuses. When trouble occurs in an apparatus and such apparatus is separated from the mutual monitoring system, both can be the same apparatus. In this embodiment, the IP address of the apparatus  1 B is stored in the monitoring object address storing portion  201 A, and the IP address of the apparatus  1 D is stored in the monitor address storing portion  202 A. 
   The log collecting means  17 A has a function of collecting log information of its own apparatus  1 A at intervals of predetermined time, and additionally registering the collected log information in the own log storing portion  191 A. Log information collected by the log collecting means  17 A includes, for example, information indicating whether or not the result of detection of various kinds of sensors is abnormal. The sensor can be, for example, a temperature sensor, an earthquake sensor, or a electric voltage sensor. 
   The sending/receiving means  11 A has a function of sending and receiving data through the network  3 . 
   The monitoring means  12 A has the following functions. 
   (1) Function of sending a log acquirement request to the apparatus  1 B, which is the monitoring object apparatus of the apparatus  1 A, in a predetermined period. 
   (2) Function of registering log information sent back from the apparatus  1 B in response to the log acquirement request in the monitoring object log storing portion  192 A. 
   (3) Function of sending a trouble occurrence notification to the management server  2  and instructing the configuration changing means  14 A to start separation processing in any of the following cases: 
   a) where log information is not sent back from the apparatus  1 B when predetermined time elapses after sending the log acquirement request; and 
   b) where log information sent back indicates that a trouble has occurred in the apparatus  1 B. 
   The responding means  13 A has a function of sending to the apparatus  1 D the log information of its own apparatus  1 A which is registered in the own log storing portion  191 A when a log information acquirement request is sent from the apparatus  1 D. The apparatus  1 D is the monitor apparatus of the apparatus  1 A. 
   The configuration changing means  14 A comprises separation processing means  15 A for separating from the mutual monitoring system an apparatus in which trouble has occurred, and addition processing means  16 A for adding a new apparatus to the mutual monitoring system. 
     FIG. 3  is a block diagram showing an example of the configuration of the separation processing means  15 A, which comprises searching means  151 A, management information changing means  152 A, log information changing means  153 A and separation notifying means  154 A. 
   The searching means searches the search target apparatus, which is the apparatus monitored by the apparatus in which trouble has occurred. For example, when a trouble occurs in the apparatus  1 B which is a monitoring object apparatus of the apparatus  1 A, the searching means  151 A searches the apparatus  1 C which has been monitored by the apparatus  1 B. 
   The management information changing means  152 A has the following functions. 
   (1) Function of changing the monitoring object apparatus from the apparatus  1 B to the apparatus  1 C by changing the IP address of the apparatus  1 B registered in the monitoring object address storing portion  201 A to the IP address of the apparatus  1 C searched out by the searching means  151 A. 
   (2) Function of sending to a new monitoring object apparatus (apparatus  1 C) a first address change request for requesting the change of the IP address of the monitor apparatus to the IP address of its own apparatus  1 A. 
   (3) Function of changing the contents of the monitor address storing portion  202 A in accordance with the aforementioned request when the first address change request is sent from another apparatus. 
   The log information changing means  153 A has the following functions. 
   (1) Function of sending log information registered in the monitoring object log storing portion  192 A to the management server  2 . 
   (2) Function of sending to a new monitoring object apparatus (apparatus  1 C) a first log information request for requesting log information of the apparatus  1 C. 
   (3) Function of registering (overwriting) in the monitoring object log storing portion  192 A log information sent from the apparatus IC in response to the first log information request. 
   (4) Function of sending log information of its own apparatus  1 A registered in the own log storing portion  191 A to a requester when the first log information request is sent from another apparatus. 
   The separation notifying means  154 A has a function of notifying the management server  2  that the apparatus  1 B in which trouble has occurred has been separated. 
     FIG. 4  is a block diagram showing an example of the configuration of the addition processing means  16 A, which comprises selection means  161 A, management information managing means  162 A, log information managing means  163 A and incorporation notifying means  164 A. 
   The selection means  161 A has a function of selecting two apparatuses from the apparatuses constituting the mutual monitoring system. Of the two apparatuses, one is to be the monitoring object apparatus of the apparatus  1 A and the other is to be the monitor apparatus of the apparatus  1 A when its own apparatus  1 A is newly added to the mutual monitoring system. 
   The management information managing means  162 A has the following functions. 
   (1) Function of registering in the monitoring object address storing portion  201 A and the monitor address storing portion  202 A the IP addresses of the monitoring object apparatus and the monitor apparatus which have been selected by the selection means  161 A, respectively. 
   (2) Function of sending a second address change request for requesting the change of the IP address of the monitor apparatus to the IP address of its own apparatus  1 A to the monitoring object apparatus selected by the selection means  161 A. 
   (3) Function of changing the contents of the monitor address storing portion  202 A in accordance with the aforementioned second address change request when the second address change request is sent from another apparatus. 
   (4) Function of sending a third address change request for requesting the change of the IP address of the monitoring object apparatus to the IP address of its own apparatus  1 A to the monitor apparatus selected by the selection means  161 A. 
   (5) Function of changing the contents of the monitoring object address storing portion  201 A in accordance with the aforementioned third address change request when the third address change request is sent from another apparatus. 
   The log information managing means  163 A has the following functions. 
   (1) Function of sending to the monitoring object apparatus selected by the selection means  161 A a second log information request for requesting log information of the apparatus. 
   (2) Function of registering in the monitoring object log storing portion  192 A log information sent from the aforementioned monitoring object apparatus in response to the second log information request. 
   (3) Function of sending log information registered in the own log storing portion  191  to a requester when the second log information request is sent from another apparatus. 
   The incorporation notifying means  164  has a function of notifying the management server  2  that its own apparatus  1 A has been incorporated in the mutual monitoring system. 
   The apparatus, for example apparatus  1 A having these functions can be implemented with a computer and software. It is implemented with a computer in, for example, a manner described below. A disk, semiconductor memory or other recording medium storing a program for making a computer function as the apparatus  1 A is prepared, and the computer is made to read the aforementioned program. The computer functions as the sending/receiving means  11 A, the monitoring means  12 A, the responding means  13 A, the configuration changing means  14 A and the log collecting means  17 A by controlling operations of itself in accordance with instructions in the read program. 
   Other apparatuses have a configuration similar to that of the apparatus  1 A, and can be implemented with a computer and software. 
   Operations of this embodiment will now be described in detail. 
   First, monitoring operations performed in each of the apparatuses  1 A to  1 D will be described. In the exemplary embodiment, the apparatuses  1 A,  1 B,  1 C and  1 D monitor the apparatuses  1 B,  1 C,  1 D and  1 A, respectively, as shown in  FIG. 5A . In this case, the IP addresses of the apparatuses  1 B,  1 C,  1 D and  1 A are registered in the monitoring object address storing portions  201 A,  201 B,  201 C and  201 D in the apparatuses  1 A,  1 B,  1 C and  1 D, respectively, and the IP addresses of the apparatuses  1 D,  1 A,  1 B and  1 C are registered in the monitor address storing portions  202 A,  202 B,  202 C and  202 D in the apparatuses  1 A,  1 B,  1 C and  1 D, respectively. 
   Monitoring means  12 A to  12 D in the apparatuses  1 A to  1 D perform processing shown in the flowchart of  FIG. 6  at intervals of predetermined period. The processing is as follows if explained taking the monitoring means  12 A as an example. 
   The monitoring means  12 A first inputs the IP address of the monitoring object apparatus  1 B from the monitoring object address storing portion  201 A, and sends a log acquirement request to the apparatus  1 B identified with the IP address inputted (steps S 61  and S 62 ). 
   When the log acquirement request is sent from the apparatus  1 A, the responding means  13 B in the apparatus  1 B inputs log information to be sent from the own log storing portion  191 B, and sends back the log information to the requester which is the apparatus  1 A (steps S 71  and S 72  in  FIG. 7 ). This log information includes information indicating states of various kinds of sensors as described previously. 
   When receiving log information within a predetermined time after sending the log acquirement request (Yes in step S 63  and No in step S 64  in  FIG. 6 ), the monitoring means  12 A of the apparatus  1 A additionally registers the received log information in the monitoring object log storing portion  192 A (step S 65 ). Thereafter, it is checked whether or not the log information indicates that a trouble has occurred in the apparatus lB, and if the log information indicates that no trouble has occurred (No in step S 66 ), the processing is ended. If the log information sent from the apparatus  1 B indicates occurrence of a trouble (Yes in step S 66 ), the management server  2  is notified that a trouble has occurred in the apparatus  1 B (step S 67 ). Next, the separation processing means  15 A is instructed to start processing (step S 68 ). If log information is not received within a predetermined time after the log acquirement request is sent (Yes in step S 64 ), processing of steps S 67  and S 68  is performed. 
   Operations when trouble occurs in the apparatus will now be described taking as an example a case where trouble occurs in the apparatus  1 B. 
   When trouble occurs in the apparatus  1 B [(a) in  FIG. 5A ], the monitoring means  12 A in the apparatus  1 A which monitors the apparatus  1 B detects the occurrence of a trouble (Yes in step S 64  or step S 66  in  FIG. 6 ), and notifies the management server  2  that a trouble has occurred in the apparatus  1 B [(b) in  FIG. 5A  and step S 67  in  FIG. 6 ]. Next, the separation processing means  15 A is instructed to start processing (step S 68 ). 
   When instructed to start processing by the monitoring means  12 A, the separation processing means  15 A first uses the searching means  151 A to search an apparatus that has used as a monitor apparatus of the apparatus  1 B in which a trouble has occurred [(c) in  FIG. 5A ]. In this case, the apparatus  1 C is searched. In other words, it is the search target apparatus. 
   The searching means  151 A performs processing shown in, for example, the flowchart of  FIG. 8  for searching a search target apparatus, which is, in this case, apparatus  1 C. 
   First, the searching means  151 A inputs the IP address of the apparatus  1 D from the monitor address storing portion  202 A. The apparatus  1 D is monitoring the apparatus  1 A and it is the monitor apparatus of apparatus  1 A. And then the searching means  151 A sends an inquiry about the address of the monitor apparatus of the apparatus  1 D to the apparatus  1 D which is identified with the inputted IP address (steps S 81  and S 82 ). Thereafter, the searching means  151 A goes into a waiting state of receiving the response from the apparatus  1 D (step S 83 ). 
   When the inquiry about the address of the monitor apparatus is sent from the apparatus  1 A, searching means  151 D in the apparatus  1 D inputs the IP address of the apparatus  1 C monitoring its own apparatus  1 D from a monitor address storing portion  202 D in its own apparatus  1 D, and sends back the IP address to the inquirer, that is to the apparatus  1 A (steps S 91  and S 92 ) as shown in the flowchart of  FIG. 9 . 
   When receiving the IP address of the apparatus  1 C sent from the apparatus  1 D (step S 83  in  FIG. 8 ), the searching means  151 A of an inquirer (apparatus  1 A) compares the received IP address with the IP address registered in the monitoring object address storing portion  201 A (IP address of the apparatus  1 B which is monitored by its own apparatus  1 A) (step S 84 ). In this example, the result of comparison shows that the former is different with the latter (No in step S 85 ), and therefore the searching means  151 A sends an inquiry about the address of the monitor apparatus of the apparatus  1 C to the apparatus  1 C based on the received IP address (step S 86 ). 
   When the inquiry about the address of the monitor apparatus is sent from the apparatus  1 A, searching means  151 C in the apparatus  1 C inputs from a monitor address storing portion  202 C the IP address of the apparatus  1 B by which its own apparatus  1 C is monitored, and sends back the IP address to the inquirer, which is the apparatus  1 A (steps S 91  and S 92  in  FIG. 9 ). 
   Since the IP address of the apparatus  1 B received this time is same with the IP address of the apparatus  1 B as the monitoring object apparatus of its own apparatus, registered in the monitoring object address storing portion  201 A (Yes in step S 85 ), the searching means  151 A in the apparatus  1 A retains therein the IP address of the sender (the apparatus  1 C) as the IP address of a search target apparatus, which is to be a new monitoring object apparatus, and then instructs the log information changing means  153 A to start processing (steps S 87  and S 88 ). 
   When instructed to start processing, the log information changing means  153 A sends log information registered in the monitoring object log storing portion  192 A (which is the log information of the apparatus  1 B) to the management server  2  [(d) in  FIG. 5B  and step S 101  in  FIG. 10 ]. Thereafter, the log information changing means  153 A sends a first log information request to the apparatus  1 C as a new monitoring object apparatus, and goes into a waiting state for receiving the response (step S 102 ). 
   When receiving the first log information request from the apparatus  1 A, log information changing means  153 C in the apparatus  1 C inputs log information of its own apparatus  1 C from the own log storing portion  191 C, and sends back that log information to the requester, that is the apparatus  1 A [steps S 111  and S 112  in  FIG. 11  and (e) in  FIG. 5B ]. 
   When log information is sent from the apparatus  1 C as the new monitoring object apparatus, the log information changing means  153 A in the apparatus  1 A registers the log information in the monitoring object log storing portion  192 A, and then instructs the management information changing means  152 A to start processing (steps S 103  and S 104  in  FIG. 10 ). 
   When instructed to start processing, the management information changing means  152 A replaces the IP address of the apparatus  1 B registered in the monitoring object address storing portion  201 A with the IP address of the apparatus  1 C as the new monitoring object apparatus (step S 121  in  FIG. 12 ). Thereafter, the management information changing means  152 A sends a first address change request including the IP address of its own apparatus  1 A to the apparatus  1 C as the new monitoring object apparatus [step S 122  and (f) in  FIG. 5B ], and ends processing. 
   When receiving the first address change request, the management information changing means  152 C in the apparatus  1 C replaces the IP address of the apparatus  1 B stored in the monitor address storing portion  202 C with the IP address of the new monitor apparatus (the apparatus  1 A) included in the first address change request (step S 131 ) as shown in the flowchart of  FIG. 13 . 
   When processing by the management information changing means  152 A is ended, the separation means  154 A in the apparatus  1 A sends to the management server  2  a separation notification indicating that the apparatus  1 B has been separated from the mutual monitoring system[(g) in  FIG. 5B ]. Thereafter, the apparatus  1 A will monitor the apparatus  1 C as the monitoring object apparatus instead of the apparatus  1 B in which trouble had occurred. When the apparatus  1 B recovers, it is incorporated into the mutual monitoring system according to the processes for incorporating a new apparatus as described below. 
   Operations for incorporating a new apparatus, or a recovered apparatus, in the mutual monitoring system will now be described. The operations will be described taking as an example a case where a new apparatus  1 E is incorporated in the mutual monitoring system as shown in  FIGS. 14A and 14B . 
   When the apparatus  1 E is to be incorporated in the mutual monitoring system, a manager of the apparatus  1 E inputs an addition processing start instruction from an input portion  21 E. 
   When the addition processing start instruction is inputted from the input portion  21 E, the addition processing means  16 E in the apparatus  1 E selects a position in which the apparatus  1 E is to be added using selection means  161 E [(h) in  FIG. 14A ]. 
   The selection means  161 E performs processing shown in, for example, the flowchart of  FIG. 15 . 
   First, the selection means  161 E sends an IP address inquiry by broadcast to inquire of the apparatuses  1 A to  1 D constituting the mutual monitoring system about their IP addresses (step S 151 ). Thereafter, the selection means  161 E goes into a waiting state for receiving the response (step S 152 ). 
   When receiving the IP address inquiry from the apparatus  1 E, selection means  161 A to  161 D in the apparatuses  1 A to  1 D send back the IP addresses of its own apparatuses  1 A to  1 D to the apparatus  1 E of the inquirer (step S 161  in  FIG. 16 ). 
   The selection means  161 E in the inquirer, which is the apparatus  1 E, waits until a predetermined time (time sufficient for acquiring the IP addresses from all apparatuses  1 A to  1 D constituting the mutual monitoring system) elapses after sending the IP address inquiry. Thereafter, the selection means  161 E selects the monitoring object apparatus and the monitor apparatus for the apparatus  1 E from the received IP addresses of apparatuses  1 A to  1 D. The two addresses are those of two apparatuses of which one is going to be the monitor apparatus of the apparatus  1 E, monitors the other, and which one is to be the monitoring object apparatus of the apparatus  1 E. For example, two IP addresses of which the host part (host address) is closest to the host part of the IP address of its own apparatus E are selected, the apparatus indicated by one IP address (e.g. higher address) is selected as an apparatus to be monitored by the apparatus  1 E (monitoring object apparatus for apparatus  1 E), and the apparatus indicated by the other IP address is selected as an apparatus to monitor the apparatus  1 E (monitor apparatus for the apparatus  1 E) (step S 153  in  FIG. 15 ). In this example, the apparatuses  1 B and  1 A are selected as a monitoring object apparatus and a monitor apparatus, respectively. 
   Thereafter, the selection means  161 E instructs log information managing means  163 E to start processing (step S 154 ). 
   When instructed to start processing, the log information managing means  163 E sends a second log information request to the apparatus  1 B as the monitoring object apparatus, and goes into a response waiting state (step S 171  in  FIG. 17 ). 
   When receiving the second log information request, the log information managing means  163 B in the apparatus  1 B inputs log information from the own log storing portion  191 B, and sends back the log information to the log information managing means  163 E of a requester, which is the apparatus  1 E [steps S 181  and S 182  in  FIG. 18  and (i) in  FIG. 14A ]. 
   The log information managing means  163 E of a requester registers in the monitoring object log storing portion  192 E the log information sent from the apparatus  1 B as the monitoring object apparatus (step S 172  in  FIG. 17 ), and then instructs the management information managing means  162 E to start processing (step S 173 ). 
   When instructed to start processing, the management information managing means  162 E sends second and third address change requests to the apparatus  1 B as the monitoring object apparatus and the apparatus  1 A as the monitor apparatus, respectively (step S 191 ), then stores the IP address of the apparatus  1 B as the monitoring object apparatus in the monitoring object address storing portion  201 E, stores the IP address of the apparatus  1 A as the monitor apparatus in the monitor address storing portion  202 E (step S 192 ), and ends processing as shown in the flowchart of  FIG. 19 . The second and third address change requests include the IP address of the apparatus  1 E. 
   When receiving the second address change request, the management information managing means  162 B in the apparatus  1 B rewrites the contents of the monitor address storing portion  202 B with the IP address of the apparatus  1 E included in the aforementioned request [step S 201  in  FIG. 20  and (j) in  FIG. 14A ]. When receiving the third address change request, the management information managing means  162 A in the apparatus  1 A rewrites the contents of the monitoring object address storing portion  201 A with the IP address of the apparatus  1 E included in the aforementioned request [step S 211  in  FIG. 21  and (k) in  FIG. 14A ]. By the above processing, the apparatus  1 E is incorporated in the mutual monitoring system [(m) in  FIG. 14B ], and incorporation notifying means  164 E then notifies the management server  2  that the apparatus  1 E has been incorporated in the mutual monitoring system. This notification includes, for example, the IP address of the incorporated apparatus  1 E, the IP address of the apparatus  1 B as the monitoring object apparatus of the apparatus  1 E, and the IP address of the apparatus  1 A as the monitor apparatus of the apparatus  1 E. 
   In the explanations described above, the searching means searches for a new monitoring object apparatus by unicast if a trouble occurs in an apparatus, but the searching means may search by broadcast. In this case, the searching means  151  performs processing shown in the flowchart of  FIG. 22  instead of processing shown in the flowchart of  FIG. 8 . In the explanations described below, the mutual monitoring system is comprised of apparatuses  1 A to  1 D as shown in  FIG. 5A , and trouble occurs in the apparatus  1 B. 
   Referring to  FIGS. 22 and 5A  and  5 B, the searching means  151 A first sends an inquiry about the address of a monitor apparatus to each of the apparatuses constituting the mutual monitoring system by broadcast (step S 221 ). This broadcast may be done by passing the inquiry through the apparatuses in the system following the monitoring object address and monitor address in each apparatus. 
   Searching means  151 C and  151 D in the apparatuses  1 C and  1 D which operate in normal condition input the IP addresses of the apparatuses  1 B and  1 C as the monitor apparatuses from the monitor address storing portions  202 C and  202 D in its own apparatuses  1 C and  1 D, respectively, and send back the IP addresses to the requester, which is the apparatus  1 A (steps S 91  and S 92  in  FIG. 9 ). 
   The searching means  151 A resisters the IP address of the monitor apparatus in a list (not shown) each time it receives the IP address (Yes in step S 222  in  FIG. 22 , step S 223 ). At this time, the IP address of the sender is registered at the same time. 
   When a predetermined time elapses after sending the inquiry about a monitoring object (Yes in step S 224 ), the searching means  151 A gives attention to one of the IP addresses of the monitor apparatuses registered in the list, and compares this IP address with the IP address of the apparatus  1 B which is the monitoring object apparatus by its own apparatus  1 A (step S 225 , Yes in step S 226 , and step S 227 ). 
   If the former is different from the latter (No in step S 228 ), processing returns to processing of step S 225 . If the former is same as the latter (Yes in step S 228 ), the IP address of an apparatus from which the IP address given attention is sent back and is retained inside as the IP address of an apparatus that is a new monitoring object apparatus (step S 229 ), and the log information changing means  153 A is then instructed to start processing (step S 230 ). Subsequent processing is as described previously. If none of the IP addresses of the monitor apparatuses registered in the list is same as the IP address of the apparatus lB, which its own apparatus  1 A monitors as the monitoring object apparatus (No in step S 226 ), the searching means displays the fact on a display portion  22 A as an error in processing(step S 231 ), and the processing is ended.