Abstract:
A computer system for realizing increased speed of identifying extent of a failure in a messaging system, provided with: a first computer including a message receiving part, a first log output part, and a first memory part configured to store receiving log data; a second computer including a data store management part configured to manage a data store, a first search part configured to search a message that meets a given condition from among messages stored in the data store, a second log output part, and a second memory part configured to store data store log data; a third computer including a message sending part, a third log output part, and a third memory part configured to store sending log data; and a fourth computer including a monitoring part, a log collecting part, and a second search part configured to search for lost message.

Description:
INCORPORATION BY REFERENCE 
       [0001]    The present application claims priority from Japanese patent application JP 2013-110830 filed on May 27, 2013, the content of which is hereby incorporated by reference into this application. 
       BACKGROUND 
       [0002]    The subject matter disclosed herein relates to a system and method capable of identifying the extent of a failure quickly in a messaging system. 
         [0003]    In recent years, a huge amount of data (big data) is sent/received over a network nowadays, and the importance of a messaging technology for processing big data efficiently is rising. With a common messaging technology, processing of relaying a message (for example, sensor data) to a destination server is executed by using a messaging system that includes a message receiving server, a message sending server, and backup storage where messages are stored. 
         [0004]    The messaging system executes processing described below in the message relaying processing. 
         [0005]    First, the receiving server receives a message from a sending terminal, stores the received message in the backup storage, and then sends a response to the sending terminal. Thereafter, the sending server obtains the message from the backup storage, relays the obtained message to a destination server, and then deletes the message from the backup storage. 
         [0006]    By following the processing steps described above, a response to the sending terminal can be sent as soon as the storing of the message in the backup storage is finished, and the response performance is accordingly improved. Storing a message in the backup storage also guarantees the permanence of the stored message. 
         [0007]    In recent years, messaging systems capable of processing a large amount of data efficiently and quickly by utilizing in-memory Key Value Store (KVS) are becoming popular. 
         [0008]    In-memory KVS is a form of data store where data that is a pair of a key and a value is stored in a volatile memory. With in-memory KVS, high scalability is accomplished and, because data is stored on the memory, high processing performance is achieved as well. In-memory KVS is also capable of preventing data loss, which is a concern in an in-memory environment, through the replication (duplication) of data between a plurality of servers. 
         [0009]    In the following description, processing of storing data that is a pair of a key and a value in a volatile memory is referred to as storing, processing of reading a value that is associated with a key out of the volatile memory is referred to as retrieval, and processing of erasing data that is a pair of a key and a value from the volatile memory is referred to as deletion. 
         [0010]    In a messaging system that utilizes the in-memory KVS described above, data is managed on the in-memory (volatile memory) for quick processing, which has a possibility of complete data loss in the event of multiple failures. Further, when a data loss occurs, identifying the lost data is difficult because the receiving server, the sending server, and an in-memory KVS all handle different aspects of message processing from one another. The resulting problem is that grasping the extent of a failure (how many users have been affected by the message loss, how many messages have been lost, and the like) is difficult. 
         [0011]    When a data loss occurs in a messaging system and a sending failure due to the data loss cannot be notified to the sender side, the failure is generally treated as a service failure. Accordingly, when dealing with a data loss by restoring lost data, or by identifying the extent of the failure and sending an error response to the sending terminal, is not possible in a messaging system that utilizes in-memory KVS, there is a chance of service failure, which makes the messaging system an unstable system. 
         [0012]    A known method of preventing a service failure by restoring data is a technology described in U.S. Pat. No. 4,159,517 A. 
         [0013]    The technology described in U.S. Pat. No.  4 , 159 , 517  A is commonly known as Write Ahead Logging (WAL), which is a DB technology. Write Ahead Logging is a technology used in in-memory DBs, typically, Hbase, and involves recording the specifics of processing in a non-volatile storage medium as a log when a piece of data is written so that the piece of data is restored by re-executing the processing recorded in the log. In Write Ahead Logging, each time the receiving server and the sending server execute processing that is related to in-memory KVS, the in-memory KVS records in a non-volatile storage medium a log of the processing including the specifics of the stored data. In the case where a failure occurs in the in-memory KVS, the in-memory KVS is reactivated, and the processing recorded in the log is executed to recover to the state that precedes the failure. By following these steps, the system can return to the state prior to data loss and Write Ahead Logging is thus capable of preventing a service failure due to data loss. 
         [0014]    A known method of preventing a service failure by ensuring that the extent of a failure can be identified is a technology described in JP 2004-227360 A. 
         [0015]    JP 2004-227360 A includes the following description: “Session information, which is assigned to each execution of processing in log information held by a server, is recorded in a session information management table, and a session information association relation between different pieces of log information is recorded in a session information association table. The session information association table is searched recursively for a piece of session information that is specified by a user through a display target entering part, to thereby identify a series of pieces of session information related to processing that is of interest to the user. For each piece of session information out of the series of related pieces of session information, an associated piece of information is collected from the log information. The collected pieces of information are integrated by the date of recording of the log and formatted so that it is easy to understand the flow of processing. The formatted information is then presented to the user.” 
       SUMMARY 
       [0016]    However, with the technology of U.S. Pat. No. 4,159,517 A, where a log is written in the in-memory KVS for each execution of processing and for every piece of data stored, the in-memory KVS performance drops and, consequently, the advantage of in-memory KVS which is fast processing is impaired. 
         [0017]    The technology of JP 2004-227360 A needs to check hours of logs for all receiving servers and all sending servers, and takes long to identify the extent of a failure. 
         [0018]    This invention has been made to solve the problems described above. Specifically, this invention provides a system and method capable of preventing a performance drop due to the writing of a log in an in-memory KVS, and capable of identifying the extent of a failure quickly. 
         [0019]    The present invention can be appreciated by the description which follows in conjunction with the following figures, wherein: a computer system comprises a plurality of computers to be coupled to one another through a network, and is configured to receive a plurality of messages from a terminal and send each of the plurality of messages to a destination apparatus. Each of the plurality of computers includes a processor, a memory coupled to the processor, and a network interface coupled to the processor. The computer system comprises at least one first computer, at least one second computer, at least one third computer, and at least one fourth computer. The at least one first computer includes a message receiving part configured to receive the plurality of messages from the terminal, a first log output part configured to output a plurality of receiving logs which are logs of the received plurality of messages, and a first memory part configured to store receiving log data including the plurality of receiving logs. The at least one second computer including a data store management part configured to manage a data store which is built from a storage area of the memory and stores the plurality of messages, a first search part configured to search for at least one of the plurality of messages that meets a given condition from among the plurality of messages stored in the data store, a second log output part configured to output a data store log which is a log of the searched at least one of the plurality of messages, and a second memory part configured to store data store log data including a plurality of data store logs. The at least one third computer including a message sending part configured to retrieve the plurality of messages stored in the data store and send the plurality of messages to the destination apparatus, a third log output part configured to output a plurality of sending logs which are logs of the plurality of messages sent to the destination apparatus, and a third memory part configured to store sending log data including the plurality of sending logs. The at least one fourth computer including a first monitoring part configured to monitor a state of the data store, a first log collecting part configured to obtain the receiving log data from the first memory part, the data store log data from the second memory part, and the sending log data from the third memory part, respectively, in a case where a failure in the data store is detected, a second search part configured to search for at least one of the plurality of messages which has been lost due to the failure in the data store by comparing the obtained receiving log data, data store log data, and sending log data, and a fourth memory part configured to store the receiving log data, the data store log data, and the sending log data. 
         [0020]    According to the aspect of this invention described above, only data store logs of messages that fulfill a given condition are written in the memory part, and a performance drop due to the writing of a log in the data store is thus prevented successfully. With only a small number of logs written, fast identification of the extent of a failure is accomplished as well. 
         [0021]    According to the disclosure, a performance drop due to the writing of a log may be prevented. The extent of a failure may also be identified quickly. 
         [0022]    Other objects, configurations, and effects than those described above are revealed through the following description of embodiments. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0023]    The present invention can be appreciated by the description which follows in conjunction with the following figures, wherein: 
           [0024]      FIG. 1  is an explanatory diagram for illustrating an example of the configuration of a mail system according to a first embodiment of this invention, 
           [0025]      FIG. 2  is a block diagram for illustrating an example of the configuration of an incoming mail server in the first embodiment, 
           [0026]      FIG. 3  is a block diagram for illustrating an example of the configuration of an outgoing mail server in the first embodiment, 
           [0027]      FIG. 4  is a block diagram for illustrating an example of the configuration of a data store server in the first embodiment, 
           [0028]      FIG. 5  is a block diagram for illustrating an example of the configuration of a collection/search server in the first embodiment, 
           [0029]      FIG. 6A  is an explanatory diagram for illustrating an example of the contents of receiving log data stored on the collection/search server in the first embodiment, 
           [0030]      FIG. 6B  is an explanatory diagram for illustrating an example of the contents of sending log data stored on the collection/search server in the first embodiment, 
           [0031]      FIG. 6C  is an explanatory diagram for illustrating an example of the contents of data store log data stored on the collection/search server in the first embodiment, 
           [0032]      FIG. 7  is a sequence diagram for illustrating an example of mail receiving processing of the first embodiment, 
           [0033]      FIG. 8A  and  FIG. 8B  are sequence diagrams for illustrating an example of mail sending processing of the first embodiment, 
           [0034]      FIG. 9  is a flow chart for illustrating mail data search processing that is executed by a data search processing part of the data store server in the first embodiment, 
           [0035]      FIG. 10  is a sequence diagram for illustrating processing that is executed when a failure occurs on the data store server in the first embodiment, 
           [0036]      FIG. 11  is an explanatory diagram for illustrating log data that is used by the collection/search server to identify the extent of a failure in the first embodiment, 
           [0037]      FIG. 12  is an explanatory diagram for illustrating an example of a terminal screen, which is used by the operator to operate the collection/search server of the first embodiment, 
           [0038]      FIG. 13  is a flow chart for illustrating mail data search processing that is executed by the data search processing part of the data store server in a modification example of the first embodiment, 
           [0039]      FIG. 14  is an explanatory diagram for illustrating an example of the configuration of a mail system according to a second embodiment, 
           [0040]      FIG. 15  is a block diagram for illustrating an example of the configuration of a log management server in the second embodiment, and 
           [0041]      FIG. 16  is a sequence diagram for illustrating processing that is executed in the second embodiment when a failure occurs in one of carrier networks. 
       
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
       [0042]    Embodiments of this invention are described below with reference to the accompanying drawings. 
         [0043]    The description of the embodiments uses a mail system, which is a typical messaging system. 
       First Embodiment 
       [0044]      FIG. 1  is an explanatory diagram for illustrating an example of the configuration of a mail system according to a first embodiment of this invention. 
         [0045]    The mail system, namely, messaging system, of this embodiment includes a mobile terminal  101 , a destination server  105 , an incoming mail server  106 , an outgoing mail server  107 , a data store server  108 , and a collection/search server  109 . The mail system also includes, as networks that couple the constituent apparatus of the mail system, a mobile network  102 , a carrier network  103 , and an internet  104 . 
         [0046]    The mobile terminal  101  is a terminal handled by a user or other people, and is coupled to the carrier network  103  via the mobile network  102 . The mobile network  102  is a wireless network that couples the mobile terminal  101  and the carrier network  103 . The carrier network  103  is a network that couples the mobile network  102 , the internet  104 , the incoming mail server  106 , the outgoing mail server  107 , the data store server  108 , and the collection/search server  109 . 
         [0047]    The destination server  105  is coupled to the carrier network  103  via the internet  104 , and sends/receives mail to/from the outgoing mail server  107 . 
         [0048]    The incoming mail server  106  is coupled to the carrier network  103 , receives mail sent from the mobile terminal  101  via the carrier network  103 , and stores the mail on the data store server  108 . While there is only one incoming mail server  106  in  FIG. 1 , the mail system may include a plurality of incoming mail servers  106 . 
         [0049]    The outgoing mail server  107  is coupled to the carrier network  103 , retrieves mail from the data store server  108 , and sends the mail to the destination server  105 . While there is only one outgoing mail server  107  in  FIG. 1 , the mail system may include a plurality of outgoing mail servers  107 . 
         [0050]    The data store server  108  is coupled to the carrier network  103 , and holds mail that is stored therein by the incoming mail server  106 . The data store server  108  is a message store, typically an in-memory KVS. While there is only one data store server  108  in  FIG. 1 , the mail system may include a plurality of data store servers  108 . 
         [0051]    The collection/search server  109  is coupled to the carrier network  103 , and monitors the situation of the data store server  108 . The collection/search server  109  also collects logs that are on the incoming mail server  106 , the outgoing mail server  107 , and the data store server  108 , and uses the collected logs to identify the extent of a failure. 
         [0052]    The incoming mail server  106 , the outgoing mail server  107 , the data store server  108 , and the collection/search server  109  are implemented as different apparatus in the example of  FIG. 1 . However, this embodiment is not limited thereto, and the functions of two or more of the servers may be implemented by a single apparatus. For example, a single apparatus may implement the functions of the incoming mail server  106  and the outgoing mail server  107 . To give another example, a single apparatus may implement the functions of the incoming mail server  106 , the outgoing mail server  107 , the data store server  108 , and the collection/search server  109 . 
         [0053]    Alternatively, the incoming mail server  106 , the outgoing mail server  107 , the data store server  108 , and the collection/search server  109  may be implemented with the use of a virtualization technology. For example, virtual computers corresponding to the incoming mail server  106 , the outgoing mail server  107 , the data store server  108 , and the collection/search server  109  may be generated on a single computer. 
         [0054]      FIG. 2  is a block diagram for illustrating an example of the configuration of the incoming mail server  106  in the first embodiment. 
         [0055]    The incoming mail server  106  is implemented with the use of an information processing apparatus. The hardware configuration of the incoming mail server  106  includes a processor  201 , a volatile memory  202 , a non-volatile memory  203 , and a network I/F  204 , which are coupled to one another via an internal communication line such as a bus. 
         [0056]    The processor  201  executes a program stored in the volatile memory  202 . By executing a program that is stored in the volatile memory  202  with the use of the processor  201 , the function of the incoming mail server  106  is implemented. In the following description, a program described as performing processing is actually being executed by the processor  201 . 
         [0057]    The volatile memory  202  stores a program executed by the processor  201 , and includes a storage area in which data necessary to execute the program is temporarily stored. Specifically, the volatile memory  202  stores a program that implements a mail receiving processing part  210 , and includes a volatile memory part  220 . 
         [0058]    The mail receiving processing part  210  includes a plurality of program modules, and controls receiving processing for receiving mail from the mobile terminal  101 . Specifically, the mail receiving processing part  210  includes a mail receiving part  211 , a data store server I/F  212 , and a log output part  213 . 
         [0059]    The mail receiving part  211  executes receiving processing for mail sent from the mobile terminal  101 . 
         [0060]    The data store server I/F  212  executes storing processing for storing received mail on the data store server  108 . 
         [0061]    The log output part  213  outputs, when received mail is stored on the data store server  108 , a receiving log in a data format that includes information for identifying the stored mail. The output receiving log is stored in receiving log data  231  of a non-volatile memory part  230  in this embodiment. 
         [0062]    The mail receiving part  211  may include the function of the data store server I/F  212 . 
         [0063]    The volatile memory part  220  stores data that is managed by the mail receiving processing part  210 . 
         [0064]    The non-volatile memory  203  includes the non-volatile memory part  230  in which data is to be stored permanently. 
         [0065]    The non-volatile memory part  230  stores data that is managed by the mail receiving processing part  210 , and also stores the receiving log data  231 . The receiving log data  231  is data that includes one or more receiving logs output by the log output part  213 . 
         [0066]    The network I/F  204  is an interface for coupling to other apparatus via a network. The network I/F  204  in this embodiment is coupled to the carrier network  103 . 
         [0067]    The program stored in the volatile memory  202  may instead be stored in the non-volatile memory  203 . The processor  201  in this case reads the program out of the non-volatile memory  203 , and loads the read program onto the volatile memory  202 . 
         [0068]      FIG. 3  is a block diagram for illustrating an example of the configuration of the outgoing mail server  107  in the first embodiment. 
         [0069]    The outgoing mail server  107  is implemented with the use of an information processing apparatus. The hardware configuration of the outgoing mail server  107  includes a processor  301 , a volatile memory  302 , a non-volatile memory  303 , and a network I/F  304 , which are coupled to one another via an internal communication line such as a bus. 
         [0070]    The processor  301  executes a program stored in the volatile memory  302 . By executing a program that is stored in the volatile memory  302  with the use of the processor  301 , the function of the outgoing mail server  107  is implemented. In the following description, a program described as performing processing is actually being executed by the processor  301 . 
         [0071]    The volatile memory  302  stores a program executed by the processor  301 , and includes a storage area in which data necessary to execute the program is temporarily stored. Specifically, the volatile memory  302  stores a program that implements a mail sending processing part  310 , and includes a volatile memory part  320 . 
         [0072]    The mail sending processing part  310  includes a plurality of program modules, and controls sending processing for sending mail to the destination server  105 . Specifically, the mail sending processing part  310  includes a mail sending part  311 , a data store server I/F  312 , and a log output part  313 . 
         [0073]    The mail sending part  311  executes sending processing for sending to the destination server  105  mail retrieved from the data store server  108 . 
         [0074]    The data store server I/F  312  executes retrieval processing for getting mail from the data store server  108 . 
         [0075]    The log output part  313  outputs, if send mail to the destination server  105  is completed, a sending log in a data format that includes information for identifying the sent mail. The output sending log is stored in sending log data  331  of a non-volatile memory part  330  in this embodiment. 
         [0076]    The volatile memory part  320  stores data that is managed by the mail sending processing part  310 . 
         [0077]    The mail sending part  311  may include the function of the data store server I/F  312 . 
         [0078]    The non-volatile memory  303  includes the non-volatile memory part  330  in which data is stored permanently. 
         [0079]    The non-volatile memory part  330  stores data that is managed by the mail sending processing part  310 , and also stores the sending log data  331 . The sending log data  331  is data that includes one or more sending logs output by the log output part  313 . 
         [0080]    The network I/F  304  is an interface for coupling to other apparatus via a network. The network I/F  304  in this embodiment is coupled to the carrier network  103 . 
         [0081]    The program stored in the volatile memory  302  may instead be stored in the non-volatile memory  303 . The processor  301  in this case reads the program out of the non-volatile memory  303 , and loads the read program onto the volatile memory  302 . 
         [0082]      FIG. 4  is a block diagram for illustrating an example of the configuration of the data store server  108  in the first embodiment. 
         [0083]    The data store server  108  is implemented with the use of an information processing apparatus. The hardware configuration of the data store server  108  includes a processor  401 , a volatile memory  402 , a non-volatile memory  403 , and a network I/F  404 , which are coupled to one another via an internal communication line such as a bus. 
         [0084]    The processor  401  executes a program stored in the volatile memory  402 . By executing a program that is stored in the volatile memory  402  with the use of the processor  401 , the function of the data store server  108  is implemented. In the following description, a program described as performing processing is actually being executed by the processor  401 . 
         [0085]    The volatile memory  402  stores programs executed by the processor  401 , and includes a storage area in which data necessary to execute the programs is temporarily stored. Specifically, the volatile memory  402  stores programs that implement a data store management part  410  and a data search processing part  420 , and includes a volatile memory part  430 . 
         [0086]    The data store management part  410  controls data store processing procedures. The data store processing procedures include processing for storing a pair of a key and a value that is received from the incoming mail server  106  or from the outgoing mail server  107  in the volatile memory part  430 , processing for sending in response a value that is associated with a key received from the incoming mail server  106  or from the outgoing mail server  107 , and processing for deleting a value that is associated with a key received from the incoming mail server  106  or from the outgoing mail server  107 . 
         [0087]    The data search processing part  420  includes a plurality of program modules, and controls processing for searching for mail data to be output as a data store log. Specifically, the data search processing part  420  includes a data search part  421  and a log output part  422 . 
         [0088]    The data search part  421  executes processing for searching pieces of mail data  431 , which are stored in the volatile memory part  430 , for a piece of the mail data  431  that fulfills a particular condition. 
         [0089]    The log output part  422  outputs a data store log of a piece of the mail data  431  that is found through the search by the data search part  421 . The output data store log is stored in data store log data  441  of the non-volatile memory part  440  in this embodiment. 
         [0090]    The volatile memory part  430  stores data that is managed by the data store management part  410 . The volatile memory part  430  of this embodiment stores one or more pieces of the mail data  431 . The mail data  431  is data of mail stored by the data store management part  410 . The data store management part  410  in this embodiment manages the mail data  431  (a value) with a sending time and date, a unique ID, a domain name, or the like that is included in the header or other sections of the mail data  431  as a key. 
         [0091]    The non-volatile memory  403  includes the non-volatile memory part  440  in which data is stored permanently. 
         [0092]    The non-volatile memory part  440  stores data that is managed by the data store management part  410  and the data search processing part  420 . The non-volatile memory part  440  also stores the data store log data  441 . The data store log data  441  is data that includes one or more data store logs output by the log output part  422 . 
         [0093]    The network I/F  404  is an interface for coupling to other apparatus via a network. The network I/F  404  in this embodiment is coupled to the carrier network  103 . 
         [0094]    The program stored in the volatile memory  402  may instead be stored in the non-volatile memory  403 . The processor  401  in this case reads the program out of the non-volatile memory  403 , and loads the read program onto the volatile memory  402 . 
         [0095]      FIG. 5  is a block diagram for illustrating an example of the configuration of the collection/search server  109  in the first embodiment. 
         [0096]    The collection/search server  109  is implemented with the use of an information processing apparatus. The hardware configuration of the collection/search server  109  includes a processor  501 , a volatile memory  502 , a non-volatile memory  503 , and a network I/F  504 , which are coupled to one another via an internal communication line such as a bus. 
         [0097]    The processor  501  executes a program stored in the volatile memory  502 . 
         [0098]    By executing a program that is stored in the volatile memory  502  with the use of the processor  501 , the function of the collection/search server  109  is implemented. In the following description, a program described as performing processing is actually being executed by the processor  501 . 
         [0099]    The volatile memory  502  stores programs executed by the processor  501 , and includes a storage area in which data necessary to execute the programs is temporarily stored. Specifically, the volatile memory  502  stores programs that implement a failure monitoring part  510 , a log collecting part  520 , and a failure extent searching part  530 , and includes a volatile memory part  540 . 
         [0100]    The failure monitoring part  510  executes processing for monitoring a failure in the data store server  108  or other components. Specifically, the failure monitoring part  510  determines whether or not a loss of the mail data  431  stored in the volatile memory part  430  of the data store server  108  has occurred. 
         [0101]    The log collecting part  520  executes processing for obtaining logs from the respective servers and storing the obtained logs in the non-volatile memory  503 . The log collecting part  520  in this embodiment obtains the receiving log data  231 , which is stored in the non-volatile memory part  230  of the incoming mail server  106 , the sending log data  331 , which is stored in the non-volatile memory part  330  of the outgoing mail server  107 , and the data store log data  441 , which is stored in the non-volatile memory part  440  of the data store server  108 . The log collecting part  520  stores the obtained logs in a non-volatile memory part  550  of the non-volatile memory  503  as receiving log data  551 , sending log data  552 , and data store log data  553 , respectively. 
         [0102]    The failure extent searching part  530  executes processing for identifying mail that has been lost due to a failure in the data store server  108 , based on the receiving log data  551 , the sending log data  552 , and the data store log data  553 , which are stored in the non-volatile memory part  550 . 
         [0103]    The volatile memory part  540  stores data that is managed by the failure monitoring part  510 , the log collecting part  520 , and the failure extent searching part  530 . 
         [0104]    The non-volatile memory  503  includes the non-volatile memory part  550  in which data is stored permanently. 
         [0105]    The non-volatile memory part  550  stores data that is managed by the failure monitoring part  510 , the log collecting part  520 , and the failure extent searching part  530 . The non-volatile memory part  550  also stores the receiving log data  551 , the sending log data  552 , and the data store log data  553 . 
         [0106]    The receiving log data  551  includes a plurality of receiving logs obtained by the log collecting part  520  from the incoming mail server  106 . The sending log data  552  includes a plurality of sending logs obtained by the log collecting part  520  from the outgoing mail server  107 . The data store log data  553  includes a plurality of data store logs obtained by the log collecting part  520  from the data store server  108 . 
         [0107]    The network I/F  504  is an interface for coupling to other apparatus via a network. The network I/F  504  in this embodiment is coupled to the carrier network  103 . 
         [0108]    The program stored in the volatile memory  502  may instead be stored in the non-volatile memory  503 . The processor  501  in this case reads the program out of the non-volatile memory  503 , and loads the read program onto the volatile memory  502 . 
         [0109]      FIG. 6A  is an explanatory diagram for illustrating an example of the contents of the receiving log data  551  stored on the collection/search server  109  in the first embodiment.  FIG. 6B  is an explanatory diagram for illustrating an example of the contents of the sending log data  552  stored on the collection/search server  109  in the first embodiment.  FIG. 6C  is an explanatory diagram for illustrating an example of the contents of the data store log data  553  stored on the collection/search server  109  in the first embodiment. 
         [0110]    A log stored in the receiving log data  551 , a log stored in the sending log data  552 , and a log stored in the data store log data  553  each include a plurality of pieces of data in a log format that includes time and date, a unique ID, the mail address of the sender, and the subject of the mail. 
         [0111]    The unique ID is a value by which a piece of mail is always identified uniquely. By using a unique ID, the contents of mail that are included in the receiving log data  551 , the sending log data  552 , and the data store log data  553  can be associated with one another. 
         [0112]    The logs are not limited to the log format described above, and, for example, may each include the body of the mail in addition to the items of the described log format. While this increases the load of processing at the time of outputting a log, in the case where mail data that has not been sent to the destination server  105  is lost due to a failure, the lost mail can be restored with the use of the mail body included in the log and the restored mail can be sent to the destination server  105 . 
         [0113]    The logs may include information about coping processing for dealing with a failure in addition to the items of the described log format. Examples of the information about coping processing include “send a delivery failure notification to the mobile terminal” and “restore mail and send the restored mail to the destination server”. Flexible failure coping is accomplished by executing coping processing based on this information on the respective servers when a failure occurs. 
         [0114]    The specifics of processing procedures in this embodiment are described next. 
         [0115]      FIG. 7  is a diagram for illustrating processing that is executed when the incoming mail server  106  receives mail.  FIG. 8A  and  FIG. 8B  are diagrams for illustrating processing that is executed when the outgoing mail server  107  sends mail. 
         [0116]    In mail sending and receiving processing, mail, various requests, and various notifications are sent/received between the data store server I/F  212  of the incoming mail server  106  and the data store server  108 , or between the data store server I/F  312  of the outgoing mail server  107  and the data store server  108 . 
         [0117]    However, data that is actually sent/received between the incoming mail server  106  or the outgoing mail server  107  and the data store server  108  is in a data format that includes header information, a payload where the data main body is stored, or the like. 
         [0118]    In the following description, mail, various requests, and various notifications that form the payload section of actually sent/received data are used instead of the data, in order to describe a sequence focused on mail receiving processing and mail sending processing. 
         [0119]    The same applies to mail and various notifications that are sent/received between the mobile terminal  101  and the mail receiving part  211  of the incoming mail server  106 , and mail and various notifications that are sent/received between the mail sending part  311  of the outgoing mail server  107  and the destination server  105 . 
         [0120]    When processing related to the storage, retrieval, or deletion of mail in/from the volatile memory part  430  of the data store server  108  is executed, various types of management information in addition to mail may need to be sent/received between the data store server I/F  212  of the incoming mail server  106  and the data store server  108 , or between the data store server I/F  312  of the outgoing mail server  107  and the data store server  108 . 
         [0121]    In the following description, however, processing related to the storage, retrieval, or deletion of mail is described in the form of one request-one response by omitting a description about the sending/receiving of the management information, in order to describe a sequence focused on mail receiving processing and mail sending processing. 
         [0122]      FIG. 7  is a sequence diagram for illustrating an example of mail receiving processing of the first embodiment. 
         [0123]    First, the mobile terminal  101  sends mail to the incoming mail server  106  (Step S 701 ). 
         [0124]    The mail receiving part  211  of the incoming mail server  106  receives the mail sent from the mobile terminal  101  (Step S 702 ), and stores the mail (mail data) in the volatile memory part  220  (Step S 703 ). 
         [0125]    The data store server I/F  212  of the incoming mail server  106  next sends a mail storing request and the mail to the data store server  108  (Step S 704 ). 
         [0126]    Though not shown in  FIG. 7 , in the case where the mail storing request and the mail are not received properly by the data store server  108  and the subsequent processing is not executed as a result, the incoming mail server  106  notifies the mobile terminal  101  that the mail has not been sent normally. 
         [0127]    Next, the data store management part  410  of the data store server  108  receives the mail storing request and the mail (Step S 705 ), and stores the received mail (the mail data  431 ) in the volatile memory part  430  as requested by the mail storing request (Step S 706 ). After storing the mail (the mail data  431 ) in the volatile memory part  430 , the data store management part  410  of the data store server  108  sends a mail storing completion notification to the incoming mail server  106  (Step S 707 ). 
         [0128]    Though not shown in  FIG. 7 , in the case where the storing of mail (the mail data  431 ) fails in Step S 706 , the data store management part  410  notifies the incoming mail server  106  that the mail (the mail data  431 ) has not been stored. Then, the incoming mail server  106  notifies the mobile terminal  101  that the mail has not been sent normally. 
         [0129]    Next, the data store server I/F  212  of the incoming mail server  106  receives the mail storing completion notification (Step S 708 ). The log output part  213  of the incoming mail server  106  output a receiving log in a data format that includes information for identifying the mail that has been sent to the data store server  108  (Step S 709 ). The output receiving log is stored in the receiving log data  231  of the non-volatile memory part  230 . 
         [0130]    The mail receiving part  211  of the incoming mail server  106  next sends a mail sending completion notification to the mobile terminal  101  (Step S 710 ). The mobile terminal  101  receives the mail sending completion notification (Step S 711 ). 
         [0131]    After Step S 710 , the mail receiving part  211  of the incoming mail server  106  deletes from the volatile memory part  220  the mail (mail data) sent to the data store server  108  (Step S 712 ). 
         [0132]      FIG. 8A  and  FIG. 8B  are sequence diagrams for illustrating an example of mail sending processing of the first embodiment. 
         [0133]    First, the data store server I/F  312  of the outgoing mail server  107  sends a mail getting request to the data store server  108  (Step S 801 ). The mail getting request includes a key for identifying mail. 
         [0134]    The data store management part  410  of the data store server  108  receives the mail getting request (Step S 802 ), and searches the volatile memory part  430  for given mail (the mail data  431 ) as requested by the mail getting request (Step S 803 ). After finding the mail (the mail data  431 ), the data store management part  410  of the data store server  108  sends the getting completion notification and the mail to the outgoing mail server  107  (Step S 804 ). 
         [0135]    The data store server I/F  312  of the outgoing mail server  107  receives the getting completion notification and the found mail (Step S 805 ), and stores the received mail in the volatile memory part  320  (Step S 806 ). After the received mail is stored in the volatile memory part  320 , the mail sending part  311  of the outgoing mail server  107  sends this mail to the destination server  105  (Step S 807 ). 
         [0136]    The destination server  105  receives the mail sent from the outgoing mail server  107  (Step S 808 ), and sends a mail sending completion notification to the outgoing mail server  107  (Step S 809 ). 
         [0137]    The mail sending part  311  of the outgoing mail server  107  receives the mail sending completion notification from the destination server  105  (Step S 810 ). 
         [0138]    The log output part  313  of the outgoing mail server  107  outputs a sending log in a data format that includes information for identifying the mail that has been sent to the destination server  105  (Step S 811 ). The output sending log is stored in the sending log data  331  of the non-volatile memory part  330 . 
         [0139]    The log output part  313  of the outgoing mail server  107  next deletes from the volatile memory part  320  the mail (mail data) sent to the destination server  105  (Step S 812 ). The data store server I/F  312  of the outgoing mail server  107  sends a mail deleting request to the data store server  108  (Step S 813 ). 
         [0140]    The data store management part  410  of the data store server  108  receives the mail deleting request (Step S 814 ), deletes the mail (the mail data  431 ) from the volatile memory part  430  (Step S 815 ), and sends a deletion completion notification to the outgoing mail server  107  (Step S 816 ). 
         [0141]    The data store server I/F  312  of the outgoing mail server  107  receives the deletion completion notification sent from the data store server  108  (Step S 817 ). 
         [0142]      FIG. 9  is a flow chart for illustrating mail data search processing that is executed by the data search processing part  420  of the data store server  108  in the first embodiment. 
         [0143]    In the mail data search processing, mail data stored in the volatile memory part  430  is searched for a given piece of data. 
         [0144]    The data search processing part  420  executes processing described below periodically or when a given condition is fulfilled, such as when an instruction from an operator is received. 
         [0145]    The data search part  421  of the data search processing part  420  starts loop processing for the mail data  431  stored in the volatile memory part  430  (Step S 901 ). 
         [0146]    Specifically, at the time the processing is started, the data search part  421  selects one piece of the mail data  431  to be processed out of the mail data  431  stored in the volatile memory part  430 , and executes Steps S 902  to S 904  for the selected piece of the mail data  431 . Steps S 902  to S 904  are therefore executed repeatedly in the loop processing for the mail data  431  to process each of a plurality of pieces of the mail data  431  stored in the volatile memory part  430 . 
         [0147]    The data search part  421  determines whether or not the piece of the mail data  431  to be processed has been staying for a fixed length of time or longer (Step S 902 ). Specifically, the following processing is executed. 
         [0148]    The data search part  421  calculates a staying time of the mail data  431  by using the current time and time and date information that is included in the mail data  431  to be processed. 
         [0149]    For example, the staying time can be calculated as a difference between the time at which the processing is started and the sending time and date of the mail. The time and date information on the mail data  431  is included in a key so that the difference between the current time and the time included in the key can be calculated. 
         [0150]    The data search part  421  determines whether or not the calculated staying time is equal to or more than a fixed length of time. In the case where the calculated staying time is equal to or more than the fixed length of time, the data search part  421  determines that the mail data  431  to be processed has been staying for the fixed length of time or longer. The fixed length of time can be set to suit system requirements. 
         [0151]    The fixed length of time is also referred to as first time range in the following description. 
         [0152]    Step S 902  has now been described. 
         [0153]    In a case of determining that the mail data to be processed has not stayed for the fixed length of time or longer, the data search part  421  proceeds to Step S 905 . 
         [0154]    In a case of determining that the mail data to be processed has been staying for the fixed length of time or longer, the data search part  421  determines whether or not a data store log of the mail data  431  to be processed has been written onto the data store log data  441  (Step S 903 ). 
         [0155]    For example, in the case where a unique ID included in the key of the mail data  431  is utilized as a part of information on a data store log, the data search part  421  can use the unique ID included in the key to search the data store log data  441  for a data store log of the mail data  431  to be processed. 
         [0156]    In a case of determining that a data store log of the mail data  431  to be processed has been written onto the data store log data  441 , the data search part  421  proceeds to Step S 905 . 
         [0157]    In a case of determining that a data store log of the mail data  431  to be processed has not been written onto the data store log data  441 , the data search part  421  stores information on the mail data  431  to be processed in the volatile memory part  430  as data to be added to the data store log data  441  (Step S 904 ). 
         [0158]    For example, the data search part  421  records in the volatile memory part  430  the key of the mail data  431  to be added. In the case where there are a plurality of pieces of the mail data  431  to be added, the data search part  421  may generate a list of the keys of the pieces of the mail data  431  to be added, and store the list in the volatile memory part  430 . 
         [0159]    The data search part  421  next determines whether or not every piece of the mail data  431  stored in the volatile memory part  430  has been processed (Step S 905 ). 
         [0160]    In a case of determining that not every piece of the mail data  431  stored in the volatile memory part  430  has been processed, i.e., that there is at least a piece of the mail data  431  remain to be processed, the data search part  421  returns to Step S 902  to select a new piece of the mail data  431  and executes Steps S 902  to S 904 . 
         [0161]    In a case where it is determined that every piece of the mail data  431  stored in the volatile memory part  430  has been processed, the log output part  422  of the data search processing part  420  adds, to the data store log data  441  of the non-volatile memory part  440 , based on the information on the mail data  431  that is recorded in the volatile memory part  430 , the data store log of the mail data  431  to be added (Step S 906 ), and ends the whole processing. 
         [0162]    The log output part  422  in this embodiment writes a data store log that includes time and date and a unique ID in the data store log data  441 . 
         [0163]    In this embodiment, writing processing is executed for all data store logs at once after the loop processing is ended, instead of executing writing processing for a data store log of a piece of the mail data  431  to be added each time a selected piece of the mail data  431  is processed by the loop processing. A delay in data store processing due to writing processing is prevented in this manner. 
         [0164]      FIG. 10  is a sequence diagram for illustrating processing that is executed when a failure occurs on the data store server  108  in the first embodiment.  FIG. 11  is an explanatory diagram for illustrating log data that is used by the collection/search server  109  to identify the extent of a failure in the first embodiment. 
         [0165]    The failure monitoring part  510  of the collection/search server  109  sends a state checking request to the data store server  108  (Step S 1001 ). The failure monitoring part  510  may send a state checking request periodically or when an instruction from the operator is received. 
         [0166]    Step S 1001  is processing for detecting a failure in the data store server  108 . In a case where a failure in the data store server  108  is detected, the collection/search server  109  checks the chance of a loss of the mail data  431  stored in the volatile memory part  430  of the data store server  108 . 
         [0167]    The data store management part  410  of the data store server  108  receives the state checking request (Step S 1002 ), checks the running state and the like of the data store server  108 , and sends the result of the checking as state information to the collection/search server  109  (Step S 1003 ). 
         [0168]    The state information can be any kind of information as long as whether or not there is a chance of loss of the mail data  431  can be determined. For example, in the case where the mail system includes a plurality of data store servers  108 , the state information can be configuration information on the plurality of data store servers  108  and the dead/alive information on each data store server  108 . 
         [0169]    Next, the failure monitoring part  510  of the collection/search server  109  receives the state information and, based on the state information, determines whether or not there is a possibility of loss of the mail data  431  (Step S 1004 ). For example, in the case where two or more data store servers  108  are shut down, the collection/search server  109  determines that a loss of the mail data  431  is likely. 
         [0170]    In a case where it is determined that a loss of the mail data  431  is unlikely, there is no need to identify the extent of a failure, and the collection/search server  109  accordingly ends the processing without executing the subsequent steps. 
         [0171]    In a case where it is determined that there is a loss of the mail data  431 , the log collecting part  520  of the collection/search server  109  sends a log obtaining request to the incoming mail server  106 , the outgoing mail server  107 , and the data store server  108  (Step S 1005 ). In the case where the OS running on the incoming mail server  106  is a UNIX-based OS, for example, Step S 1005  is accomplished by using the scp command or the like. 
         [0172]    The incoming mail server  106 , the outgoing mail server  107 , and the data store server  108  separately send log data to the collection/search server  109  (Steps S 1006 , S 1007 , and S 1008 ). 
         [0173]    The log collecting part  520  of the collection/search server  109  receives the pieces of log data sent from the respective servers, and stores the received log data in the non-volatile memory part  550  (Step S 1009 ). The log data received from the incoming mail server  106  is stored as the receiving log data  551 , the log data received from the outgoing mail server  107  is stored as the sending log data  552 , and the log data received from the data store server  108  is stored as the data store log data  553 . 
         [0174]    The failure extent searching part  530  of the collection/search server  109  uses the received log data to identify the extent of the failure (Step S 1010 ). A method of identifying the extent of a failure is described with reference to  FIG. 11 . 
         [0175]    As described with reference to  FIG. 9 , only logs of pieces of the mail data  431  that have been staying in the mail system for a fixed length of time (the first time range) or longer are stored in the data store log data  441  in the mail data search processing. More specifically, only data store logs of pieces of the mail data  431  that precede the processing start time (the current time) by the first time range or more are stored. 
         [0176]    Therefore, in the case where the processing of  FIG. 9  and the processing of  FIG. 10  are executed in parallel, the data store log data  553  is as illustrated in  FIG. 11 . The number of logs stored in the data store log data  553  is also much smaller than the number of logs stored in the receiving log data  551  or the sending log data  552 . 
         [0177]    The extent of a failure can be identified quickly in this embodiment based on the described features of the data store log data  553 . 
         [0178]    The failure extent searching part  530  first extracts from the receiving log data  551  logs in a period between the current time and a time point that precedes the current time by the first time range. The failure extent searching part  530  also extracts from the sending log data  552  logs in the period between the current time and the time point that precedes the current time by the first time range. 
         [0179]    The time point that precedes the current time by the first time range may be referred to as first time point in the following description. 
         [0180]    In the example of  FIG. 11 , logs in the hatched portion of the receiving log data  551  and logs in the hatched portion of the sending log data  552  are extracted. 
         [0181]    The failure extent searching part  530  next extracts from the sending log data  552  logs in a period between the first time point and a time point that precedes the first time point by a length of time in which an extent of failure is to be identified. The failure extent searching part  530  also extracts from the data store log data  553  logs in the period between the first time point and the time point that precedes the first time point by the length of time in which an extent of failure is to be identified. 
         [0182]    The length of time in which a extent of failure is to be identified may be referred to as second time range and the time point that precedes the first time point by the second time range may be referred to as second time point in the following description. The second time point (the length of time in which the extent of a failure is to be identified) can be set to suit system requirements. 
         [0183]    In the example of  FIG. 11 , logs in the blackened portion of the sending log data  552  and logs in the blackened portion of the data store log data  553  are extracted. 
         [0184]    There are no duplicates between logs in the hatched portions and the logs in the blackened portions as illustrated in  FIG. 11 . This is because logs of the mail data  431  in the range from the current time to the first time point are not output. 
         [0185]    The processing of extracting the logs may be executed in a case where logs are stored in Step S 1009 . 
         [0186]    The failure extent searching part  530  compares the receiving logs in the period between the current time and the first time point with the sending logs in the period between the current time and the first time point to search for receiving logs that are not included in the sending log data  552 . For example, the failure extent searching part  530  can compare a unique ID in a receiving log with a unique ID in a sending log. 
         [0187]    The failure extent searching part  530  also compares the sending logs in the period between the first time point and the second time point with the data store logs in the period between the first time point and the second time point to search for data store logs that are not included in the sending logs. For example, the failure extent searching part  530  can compare a unique ID in a sending log with a unique ID in a data store log. 
         [0188]    The extent of a failure such as the number of pieces of the mail data  431  lost and the number of people affected by the loss of mail data is identified through the processing described above. In other words, because a log found by the search includes the sender address and the subject of the mail as illustrated in  FIG. 6A ,  FIG. 6B , and  FIG. 6C , the failure extent searching part  530  can notify information about the extent of a failure based on information that is included in the found log. 
         [0189]    Step S 1010  has now been described. 
         [0190]    Next, the failure extent searching part  530  of the collection/search server  109  sends information about the identified extent of the failure as failure extent information to the incoming mail server  106  (Step S 1011 ). For example, the found logs themselves can be sent as the failure extent information. 
         [0191]    The mail receiving part  211  of the incoming mail server  106  receives the failure extent information sent from the collection/search server  109  (Step S 1012 ), generates a delivery failure notification to be sent to the mobile terminal  101 , and sends the generated delivery failure notification to the mobile terminal  101  (Step S 1013 ). For example, a list that associates the address of a sender with the subject of mail can be sent as the delivery failure notification. 
         [0192]    The mobile terminal  101  receives the delivery failure notification sent from the incoming mail server  106  (Step S 1014 ). This informs the sender of the mail who is the owner of the mobile terminal  101  that the mail sent by the sender has failed to reach the destination apparatus. 
         [0193]    While the incoming mail server  106  deals with a failure by sending a delivery failure notification in the processing described above, this embodiment is not limited thereto. 
         [0194]    For example, in the case where the body of lost mail has been output as a log, the failure extent searching part  530  may output in Step S 1011  the found log and information about processing for dealing with a failure, restore the mail (mail data) based on the output information about processing for dealing with a failure, and send the restored mail to the destination server  105 . 
         [0195]    To accomplish the processing described above, the collection/search server  109  sends failure extent information that includes the restored mail to the outgoing mail server  107 , and the outgoing mail server  107  resends the lost mail to the destination server  105 . 
         [0196]    Examples of the processing for dealing with a failure include “sending a delivery failure notification to the mobile terminal” and “restoring mail data and sending the mail data to the destination server”. 
         [0197]    The collection/search server  109  in the processing described above automatically identifies the extent of a failure (Step S 1010 ) and sends failure extent information (Step S 1011 ). However, this embodiment is not limited thereto, and processing based on the operator&#39;s operation may be executed instead. 
         [0198]      FIG. 12  is an explanatory diagram for illustrating an example of a terminal screen, which is used by the operator to operate the collection/search server  109  of the first embodiment. 
         [0199]    A terminal screen  1201  is operated when Steps S 1010  and S 1011  of  FIG. 10  are executed. The terminal screen  1201  is a screen that is displayed by a terminal program installed as standard in a common information processing apparatus. 
         [0200]    The operator can obtain a differential between logs of different types by, for example, entering the DIFF command of the UNIX. The failure extent searching part  530  receives a given command to execute processing that resembles Step S 1010 , and displays the result of the processing on the terminal screen. 
         [0201]    The operator can also send failure extent information to the incoming mail server  106  by operating the collection/search server  109  so that a log found through a search with the use of the DIFF command is input to the failure extent searching part  530 . 
         [0202]    As described above, according to the first embodiment, only logs of pieces of mail that fulfill a given condition are written in the non-volatile memory part  440  on the data store server  108 . This makes the number of logs that are written smaller than in the related art, and a drop in the performance of data store such as in-memory KVS is accordingly prevented. 
         [0203]    In addition, the failure extent searching part  530  can identify the extent of a failure by comparing pieces of receiving log data and pieces of sending log data that are within a given time range, and comparing pieces of sending log data and pieces of data store log data that are within another given time range. 
         [0204]    Unlike the related art where the amount of receiving log data and sending log data to be compared is huge, this embodiment is capable of identifying the extent of a failure quickly by comparing pieces of data within a limited time range. Storing only logs of pieces of mail data that fulfill a given condition in the data store log also speeds up the comparison processing described above. 
       MODIFICATION EXAMPLE 
       [0205]    While a piece of the mail data  431  that is to be added to the data store log data  441  is identified based on the length of time for which the mail has been staying in the mail system in the first embodiment, the first embodiment is not limited thereto. For instance, a piece of the mail data  431  that is to be added to the data store log data  441  may be identified based on the processing described below. 
         [0206]      FIG. 13  is a flow chart for illustrating mail data search processing that is executed by the data search processing part  420  of the data store server  108  in a modification example of the first embodiment. 
         [0207]    In Modification Example 1, the data search part  421  determines whether or not the destination domain of a piece of the mail data  431  to be processed is matched with a target domain (Step S 1302 ). The target domain is set in advance. 
         [0208]    For example, in the case where a domain name included in the key of the mail data  431  is utilized as a part of information on a data store log, the data search part  421  determines whether or not the target domain is the destination domain of a piece of the mail data  431  to be processed based on the domain name included in the key. 
         [0209]    Steps S 1301 , S 1303 , S 1304 , S 1305 , and S 1306  are the same as Steps S 901 , S 903 , S 904 , S 905 , and S 906 , respectively, and a description thereof is therefore omitted. 
         [0210]    Fast failure extent identification focused on a particular domain is accomplished by executing the data search processing of  FIG. 13 . 
       Second Embodiment 
       [0211]    A mail system according to a second embodiment of this invention includes a plurality of carrier networks. In a case where a failure occurs in one of the plurality of carrier networks, the collection/search server  109  in another of the plurality of carrier networks identifies the extent of the failure. 
         [0212]    The second embodiment is described below by focusing on differences from the first embodiment. 
         [0213]      FIG. 14  is an explanatory diagram for illustrating an example of the configuration of a mail system according to the second embodiment. 
         [0214]    The mail system of the second embodiment includes two carrier networks  103 , one of which is a carrier network A and the other of which is a carrier network B. The mail system of the second embodiment also includes a log management server  1400 . The log management server  1400  is coupled to the carrier network A  103  and the carrier network B  103  to manage logs for each carrier network  103  separately. 
         [0215]    To each of the carrier network A  103  and the carrier network B  103 , the incoming mail server  106 , the outgoing mail server  107 , the data store server  108 , and the collection/search server  109  are coupled. 
         [0216]    The mobile terminal  101  of the second embodiment is coupled to the carrier network A  103  and the carrier network B  103  via the mobile network  102 . 
         [0217]    The destination server  105  of the second embodiment is coupled to the carrier network A  103  and the carrier network B  103  via the internet  104 . The destination server  105  of the second embodiment sends/receives mail to/from the outgoing mail server  107  that is coupled to the carrier network A  103 , and sends/receives mail to/from the outgoing mail server  107  that is coupled to the carrier network B  103 . 
         [0218]    The incoming mail server  106 , the outgoing mail server  107 , the data store server  108 , and the collection/search server  109  in this embodiment are the same as those in the first embodiment, and a description on the configurations of the respective servers is therefore omitted here. 
         [0219]      FIG. 15  is a block diagram for illustrating an example of the configuration of the log management server  1400  in the second embodiment. 
         [0220]    The log management server  1400  is implemented with the use of an information processing apparatus. The hardware configuration of the log management server  1400  includes a processor  1501 , a volatile memory  1502 , a non-volatile memory  1503 , and a network I/F  1504 , which are coupled to one another via an internal communication line such as a bus. 
         [0221]    The processor  1501  executes a program stored in the volatile memory  1502 . By executing a program that is stored in the volatile memory  1502  with the use of the processor  1501 , the function of the log management server  1400  is implemented. In the following description, a program described as performing processing is actually being executed by the processor  1501 . 
         [0222]    The volatile memory  1502  stores programs executed by the processor  1501 , and includes a storage area in which data necessary to execute the programs is temporarily stored. Specifically, the volatile memory  1502  stores programs that implement a failure monitoring part  1510  and a log collecting part  1520 , and includes a volatile memory part  1530 . 
         [0223]    The failure monitoring part  1510  executes processing for monitoring each carrier network  103  for a failure. 
         [0224]    The log collecting part  1520  executes processing for obtaining a log from the collection/search server  109  of each carrier network  103  and storing the obtained log in the non-volatile memory  1503 . The log collecting part  1520  in this embodiment stores, for each carrier network  103  separately, receiving log data  1541 , sending log data  1542 , and data store log data  1543  in a non-volatile memory part  1540 . 
         [0225]    The volatile memory part  1530  stores data that is managed by the failure monitoring part  1510  and the log collecting part  1520 . 
         [0226]    The non-volatile memory  1503  includes the non-volatile memory part  1540  in which data is stored permanently. 
         [0227]    The non-volatile memory part  1540  stores data that is managed by the failure monitoring part  1510  and the log collecting part  1520 . The non-volatile memory part  1540  also stores, for each carrier network  103  separately, the receiving log data  1541 , the sending log data  1542 , and the data store log data  1543 . 
         [0228]    The receiving log data  1541 , the sending log data  1542 , and the data store log data  1543  are the same as the receiving log data  551 , the sending log data  552 , and the data store log data  553 . 
         [0229]    The network I/F  1504  is an interface for coupling to other apparatus via a network. The network I/F  1504  in this embodiment is coupled to the carrier network A  103  and the carrier network B  103 . 
         [0230]      FIG. 16  is a sequence diagram for illustrating processing that is executed in the second embodiment when a failure occurs in one of the carrier networks. 
         [0231]    In the following description, the carrier network in which a failure has occurred is the carrier network A  103 . 
         [0232]    The log collecting part  1520  of the log management server  1400  sends a log obtaining request to the collection/search server  109  of each of the carrier network A  103  and the carrier network B  103  (Step S 1601 ). For example, the log management server  1400  sends a log obtaining request by broadcast. The log collecting part  1520  may send a log obtaining request periodically or may send a log obtaining request when an instruction from an operator is received. 
         [0233]    The log collecting part  520  of the collection/search server  109  of the carrier network B  103  receives the log obtaining request and sends to the log management server  1400  the receiving log data  551 , the sending log data  552 , and the data store log data  553 , which are stored in the non-volatile memory part  550  (Step S 1602 ). 
         [0234]    The log collecting part  520  may obtain the log data by sending a log obtaining request to the incoming mail server  106 , the outgoing mail server  107 , and the data store server  108  that are coupled to the carrier network B  103 . 
         [0235]    The log collecting part  1520  of the log management server  1400  receives the log data and stores the received data in the non-volatile memory part  1540  as log data of the carrier network B (Step S 1603 ). For example, the log collecting part  1520  stores the log data in the non-volatile memory part  1540  in association with the identifier of the carrier network B. 
         [0236]    The failure monitoring part  1510  of the log management server  1400  determines for each carrier network  103  whether or not a failure has occurred in the carrier network  103 , based on the carrier network&#39;s response to the log obtaining request (Step S 1604 ). 
         [0237]    For example, in the case where log data is not sent from the collection/search server  109  within a fixed length of time after the sending of the log obtaining request, the failure monitoring part  1510  determines that a failure has occurred in the carrier network  103  to which this collection/search server  109  is coupled. 
         [0238]    The failure monitoring part  1510  in this embodiment determines that a failure has occurred in the carrier network A  103  because the collection/search server  109  of the carrier network A  103  fails to send log data. 
         [0239]    In a case where a failure is detected in one of the carrier networks  103 , the failure monitoring part  1510  of the log management server  1400  sends failure information to the collection/search server  109  of the other carrier network  103  (Step S 1605 ). 
         [0240]    The failure information includes identification information on the carrier network  103  in which a failure has been detected and log data of the carrier network  103 . For example, the failure monitoring part  1510  uses identification information on the carrier network  103  in which a failure has been detected to obtain from the non-volatile memory part  1540  log data that is associated with this carrier network  103 . 
         [0241]    The log collecting part  520  of the collection/search server  109  receives the failure information (Step S 1606 ), and stores the log data included in the failure information in the non-volatile memory part  550  in association with the identification information on the carrier network  103  (Step S 1607 ). 
         [0242]    The failure extent searching part  530  of the collection/search server  109  uses the log data stored in the non-volatile memory part  550  to identify the extent of the failure (Step S 1608 ). Step S 1608  is the same as Step S 1010 , and a description thereof is therefore omitted. 
         [0243]    The failure extent searching part  530  of the collection/search server  109  next sends information about the identified failure extent to the incoming mail server  106  as failure extent information (Step S 1609 ). Step S 1609  is the same as Step S 1011 , and a description thereof is therefore omitted. 
         [0244]    The mail receiving part  211  of the incoming mail server  106  receives the failure extent information sent from the collection/search server  109  (Step S 1610 ), generates a delivery failure notification to be sent to the mobile terminal  101 , and sends the generated delivery failure notification to the mobile terminal  101  (Step S 1611 ). Steps S 1610  and S 1611  are the same as Steps S 1012  and S 1013 , respectively, and a description thereof is therefore omitted. 
         [0245]    The mobile terminal  101  receives the delivery failure notification sent from the incoming mail server  106  (Step S 1612 ). This informs the sender of the mail who is the owner of the mobile terminal  101  that the mail sent by the sender has failed to reach the destination. Step S 1612  is the same as Step S 1014 . 
         [0246]    While the incoming mail server  106  deals with a failure by sending a delivery failure notification in the processing described above, this embodiment is not limited thereto. 
         [0247]    For example, in the case where the body of lost mail has been output as a log, the log collecting part  1520  may output in Step S 1609  the found log and information about processing for dealing with a failure, restore the mail (mail data) based on the output information about processing for dealing with a failure, and send the restored mail to the destination server  105 . 
         [0248]    To accomplish the processing described above, the collection/search server  109  sends failure extent information that includes the restored mail to the outgoing mail server  107 , and the outgoing mail server  107  resends the lost mail to the destination server  105 . 
         [0249]    Examples of the processing for dealing with a failure include “sending a delivery failure notification to the mobile terminal” and “restoring mail data and sending the mail data to the destination server”. 
         [0250]    While the log management server  1400  sends a log obtaining request to the collection/search server  109  of each carrier network  103  in the processing described above, this embodiment is not limited thereto. 
         [0251]    For example, the incoming mail server  106 , the outgoing mail server  107 , and the data store server  108  may each send logs to the log management server  1400  when outputting logs (Step S 709 , S 811 , or S 906 ). The log management server  1400  in this case sends a state checking request, instead of a log obtaining request, to the collection/search server  109 . 
         [0252]    As described above, according to the second embodiment, the extent of a failure that has occurred throughout one of the carrier networks  103  can be identified by the collection/search server  109  of the other carrier network  103 . 
         [0253]    Various types of software illustrated in the present embodiment can be stored in various electromagnetic, electronic, and optical recording media and can be downloaded to a computer via a communication network such as the Internet. 
         [0254]    Further, in the present embodiment, although an example of using software-based control has been described, part of the control may be realized by hardware. 
         [0255]    While the present invention has been described in detail with reference to the accompanying drawings, the present invention is not limited to the specific configuration, and various changes and equivalents can be made within the scope of the claims.