Abstract:
A system includes a plurality of information processing apparatuses; and a management apparatus configured to: transmit Hamilton path information including a communication route information and order information indicating a position in a communication order assigned to the each of the plurality of information processing apparatuses, to each of the plurality of information processing apparatuses, and transmit a first message to one or more information processing apparatuses that are free from an abnormal condition according to the communication order, and wherein each of the plurality of information processing apparatuses is configured to: transmit a second message including information about an abnormal condition that has been detected to a next transmission destination that is free from an abnormal condition, when the first message is received, and transmit the information about an abnormal condition that has been detected to the management apparatus, when the next transmission destination fails to be identified.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2014-161251, filed on Aug. 7, 2014, the entire contents of which are incorporated herein by reference. 
       FIELD 
       [0002]    The embodiments discussed herein are related to a system and an abnormal condition detection method. 
       BACKGROUND 
       [0003]    Large-scale systems including many nodes are being implemented in recent years. A large-scale system of this type includes three types of nodes: node that executes calculation processing as commanded by the user (this node will be referred to below as the calculation node), node that operates as a file server or a database (DB) server for the calculation server (this node will be referred to below as the input-output (IO) node), and node that manages the entire system (this node will be referred to below as the management node). 
         [0004]    One of the important roles of the management node is to monitor abnormal conditions in calculation nodes and IO nodes and, if there is an abnormal condition, to execute processing to deal with the abnormal condition. In a general monitoring method, nodes to be monitored (calculation nodes and IO nodes) and a management node exchange an existence-confirming message with each other at intervals of a predetermined time. 
         [0005]    If, however, many nodes are to be monitored, the processing load on the management node becomes large. A possible way to reduce the processing load on the management node is to share the nodes to be monitored among a plurality of management nodes. It is also effective to reduce a processing load on each message. 
         [0006]    If, however, there are many nodes to be monitored, even a technology as described above is unable to be said to be sufficient in reduction of a processing load applied to the entire system to detect abnormal conditions. Related art is disclosed in, for example, Japanese Laid-open Patent Publication No. 2000-187598, Japanese Laid-open Patent Publication No. 10-049507, and International Publication Pamphlet No. WO2014/103078. 
       SUMMARY 
       [0007]    According to an aspect of the invention, a system includes a plurality of information processing apparatuses that are mutually coupled through a communication path; and a management apparatus that manages the plurality of information processing apparatuses, wherein the management apparatus is configured to: transmit Hamilton path information including a communication route information and order information indicating a position in a communication order, the position being assigned to the each of the plurality of information processing apparatuses, to each of the plurality of information processing apparatuses, and transmit a first message to one or more information processing apparatuses that are free from an abnormal condition among the plurality of information processing apparatuses, according to the communication order, and each of the plurality of information processing apparatuses is configured to: transmit a second message including information about an abnormal condition that has been detected so far to a next transmission destination that is free from an abnormal condition, when the first message is received, and transmit the information about an abnormal condition that has been detected so far to the management apparatus, when the next transmission destination fails to be identified. 
         [0008]    The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
         [0009]    It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0010]      FIG. 1  illustrates a Hamilton path; 
           [0011]      FIG. 2  also illustrates a Hamilton path; 
           [0012]      FIG. 3  illustrates an example of nodes coupled like a mesh; 
           [0013]      FIG. 4  illustrates an example of nodes coupled like a torus; 
           [0014]      FIG. 5  illustrates an example of a Hamilton path that is set for nodes coupled like a mesh; 
           [0015]      FIG. 6  illustrates an example of a Hamilton path that is set for nodes coupled like a torus; 
           [0016]      FIG. 7  illustrates an example of an information processing system according to a first embodiment; 
           [0017]      FIG. 8  is a functional block diagram of a management node according to the first embodiment; 
           [0018]      FIG. 9  is a functional block diagram of a to-be-monitored node according to the first embodiment; 
           [0019]      FIG. 10  illustrates a flow of processing executed by the management node according to the first embodiment; 
           [0020]      FIG. 11  illustrates a flow of processing executed by a to-be-monitored node according to the first embodiment; 
           [0021]      FIG. 12  illustrates a flow of processing executed by the management node according to the first embodiment; 
           [0022]      FIG. 13  illustrates an example of an information processing system according to a second embodiment; 
           [0023]      FIG. 14  is a functional block diagram of a management node according to the second embodiment; 
           [0024]      FIG. 15  is a functional block diagram of a to-be-monitored node according to the second embodiment; 
           [0025]      FIG. 16  illustrates a flow of processing executed by the management node according to the second embodiment; 
           [0026]      FIG. 17  illustrates flow of transmission processing according to the second embodiment; 
           [0027]      FIG. 18  illustrates a flow of processing executed by the management node according to the second embodiment; 
           [0028]      FIG. 19  illustrates a flow of processing executed by a reduction processing unit in the management node according to the second embodiment; and 
           [0029]      FIG. 20  is a functional block diagram of a computer. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
     First Embodiment 
       [0030]    The policy of this embodiment is to use resources of to-be-monitored nodes to reduce a load on a management node and to suppress a load on each to-be-monitored node from increasing beyond a certain level. 
         [0031]    To achieve this policy, the management node performs mutual monitoring through a Hamilton path. Specifically, to-be-monitored nodes under management by a management node are each regarded as a vertex in a graph theory and a link between each two to-be-monitored nodes is regarded as an edge in the graph theory. Links among nodes are handled as a graph. 
         [0032]    A Hamilton path is selected from this graph. A Hamilton path is a path that visits each vertex in the graph only once. If a communication route used for node monitoring is set according to this Hamilton path, it is possible to suppress the number of links used in communication and the number of communications. Therefore, it can also be expected that communication latency and interference with other communication (such as communication executed by a job) would be suppressed. 
         [0033]    In  FIG. 1 , a plurality of nodes  1   a  to  1   f  are coupled through communication paths. As illustrated in  FIG. 1 , each node is coupled directly to a plurality of other nodes, that is, there are a plurality of communication routes between each two particular nodes. A Hamilton path X, indicated by the bold arrows, is set so that it visits each of the nodes  1   a  to  1   f  only once. In the example in  FIG. 1 , the Hamilton path X visits nodes  1   a,    1   b,    1   c,    1   d,    1   e,  and  1   f  in that order. Since, however, there are a plurality of communication routes between each two particular nodes, a plurality of Hamilton paths can also be set. As illustrated in  FIG. 2 , a Hamilton path Y can also be set for the nodes  1   a,    1   b,    1   c,    1   d,    1   e,  and  1   f,  which are coupled as in  FIG. 1 . 
         [0034]    Inter-node links in an information processing system that includes many nodes often take one of two patterns called “mesh” and “torus”. In the case of two dimensions, a mesh is formed by setting nodes and inter-node links like a grid, as illustrated in  FIG. 3 . Also in the case of two dimensions, if at least one wraparound link is added to a mesh between an upper-end node and a lower-end node or between a left-end node and a right-end node as illustrated in  FIG. 4 , the pattern is a torus. 
         [0035]    A Hamilton path as illustrated in  FIG. 5  can be selected from a mesh as illustrated in  FIG. 3 . Another Hamilton path can also be selected. In the case of three dimensions, a Hamilton path can be similarly selected in a plane linked from the end point in the immediately below plane. If this is repeated for the next dimension, a Hamilton path can be selected from an N-dimensional mesh as well. 
         [0036]    A Hamilton path as illustrated in  FIG. 6  can be selected from a torus as illustrated in  FIG. 4 . If the torus is expanded as with a mesh, a Hamilton path can be selected in an N-dimensional torus as well. In the case of an N-dimensional torus, a Hamilton closed path, the start point and end point of which are the same, can also be selected. 
         [0037]    The structure of a system will be described with reference to  FIGS. 7 to 9 , in which, to monitor target nodes, a management node sends a monitoring message through a communication route that is set to monitor these nodes along a Hamilton path as described above. 
         [0038]      FIG. 7  outlines an information processing system. The information processing system includes a management node  200 , a to-be-monitored node group  100  (including to-be-monitored nodes  1   a  to  1   f ). The management node  200  and each to-be-monitored node  1  in the node group  100  are mutually coupled through, for example, a local area network (LAN). 
         [0039]      FIG. 8  is a functional block diagram of the management node  200 . The management node  200  includes an initialization unit  210 , a Hamilton path data storage unit  220 , a monitoring processing unit  230 , a communication unit  240 , a memory  250 , and an abnormal condition processing unit  260 . 
         [0040]    The initialization unit  210  includes a Hamilton path creating unit  211 . The initialization unit  210  sets Hamilton path data for each to-be-monitored node  1  in the node group  100  through the communication unit  240 . The Hamilton path data created by the Hamilton path creating unit  211  is stored in the Hamilton path data storage unit  220 . 
         [0041]    The monitoring processing unit  230  causes the communication unit  240  to send, for example, a message that flows through the Hamilton path to count the number of abnormal nodes. 
         [0042]    The communication unit  240  includes a remote direct memory access (RDMA) unit  241 . The RDMA unit  241  accepts RDMA from an RDMA unit in a to-be-monitored node  1  and writes to the memory  250 . In RDMA, the to-be-monitored node  1  carries out a transfer to the memory  250  without involving the operating system (OS) in the management node  200 , enabling high-throughput communication with low latency. A processing load on the management node  200  can also be reduced. 
         [0043]    The memory  250  includes a summary data area  251  and a node data area  252 . The summary data area  251  stores summary data about abnormal conditions detected on the Hamilton path such as, for example, the total of abnormal nodes. The node data area  252  stores flags, each of which indicates whether the relevant to-be-monitored node  1  is abnormal, and the like. 
         [0044]    The abnormal condition processing unit  260  executes abnormal condition handing processing according to the summary data and node data stored in the memory  250 . 
         [0045]      FIG. 9  is a functional block diagram of the to-be-monitored node  1 . The to-be-monitored node  1  includes a communication unit  10 , a monitoring processing unit  20 , and a Hamilton path data storage unit  30 . 
         [0046]    The communication unit  10  includes an RDMA unit  11 . The communication unit  10  communicates with another to-be-monitored node  1  or the management node  200 . The RDMA unit  11  directly transfers data to the memory  250  through the RDMA unit  241  in the management node  200 . 
         [0047]    The monitoring processing unit  20  executes processing in relation to a message received at the communication unit  10 . The monitoring processing unit  20  sends a message to the other to-be-monitored node  1  through the communication unit  10  or causes the RDMA unit  11  to send data to the management node  200 . 
         [0048]    The Hamilton path data storage unit  30  stores data about a Hamilton path that has been set. 
         [0049]    Next, processing executed by the information processing system according to this embodiment will be described with reference to  FIGS. 10 to 12 . In this embodiment, the nodes in node groups  100  are coupled in an N-dimensional mesh. 
         [0050]    First, the Hamilton path creating unit  211  in the initialization unit  210  creates a Hamilton path according to a certain algorithm and stores the created Hamilton path in the Hamilton path data storage unit  220  (S 1  in  FIG. 10 ). The certain algorithm may be any of conventional algorithms, so its description will be omitted here. A Hamilton path may be given by the manager in advance, in which case processing in S 1  is skipped. 
         [0051]    Next, the initialization unit  210  causes the communication unit  240  to notify each to-be-monitored node  1  of Hamilton path data including a communication route and a serial number (S 3 ). It is desirable for a node, on the Hamilton path, next to the local node to be operating normally. If the next node is not operating normally, processing to search for a node that is operating normally is executed as described below. Therefore, the initialization unit  210  distributes data about nodes on the Hamilton path that are behind at least the node that is the distribution destination. 
         [0052]    After that, the monitoring processing unit  230  initializes an err_num counter, which indicates the number of abnormal nodes, to 0, the counter indicating the number of abnormal nodes, and also initializes an n counter to 0, the counter indicating the node number (serial number) of the to-be-monitored node string on the Hamilton path (S 5 ). 
         [0053]    Next, the monitoring processing unit  230  causes the communication unit  240  to send a message including the value of the err_num counter to the to-be-monitored node  1  with the serial number n (S 7 ). The communication unit  10  in the to-be-monitored node  1  receives the message and replies with, for example, an acknowledgement (ACK) message. 
         [0054]    The monitoring processing unit  230  determines whether the communication unit  240  has received an ACK message from the to-be-monitored node  1  with the serial number n within a predetermined time, that is, a time-out has occurred (S 9 ). If the monitoring processing unit  230  determines that a time-out has not occurred, that is, an ACK message has been received from the to-be-monitored node  1  with the serial number n within the predetermined time, the processing by the management node  200  at this stage is terminated. 
         [0055]    If the monitoring processing unit  230  determines that a time-out has occurred, that is, an ACK message has not been received from the to-be-monitored node  1  with the serial number n within the predetermined time, the monitoring processing unit  230  indicates an error in an area, in the node data area  252 , that is reserved for the serial number n (S 11 ). For example, the monitoring processing unit  230  turns on a relevant error flag. 
         [0056]    The monitoring processing unit  230  then increments the err_num counter by one and also increments the n counter by one (S 13 ). The monitoring processing unit  230  determines whether the n counter has reached a to-be-monitored node count n max  (S 15 ). That is, the monitoring processing unit  230  determines whether a message has been sent from the management node  200  to all to-be-monitored nodes  1 . 
         [0057]    If the monitoring processing unit  230  determines that the n counter has not reached the to-be-monitored node count n max , the process returns to S 7 , where the monitoring processing unit  230  sends a message to a next to-be-monitored node  1  on the Hamilton path. 
         [0058]    If the monitoring processing unit  230  determines that the n counter has reached the to-be-monitored node count n max , this indicates that none of the to-be-monitored nodes  1  is operating normally. In this case, the monitoring processing unit  230  writes the value of the err_num counter, which indicates the number of abnormal nodes, the value being the to-be-monitored node count n max , to the summary data area  251  (S 17 ). Then, the process is terminated. 
         [0059]    As described above, the monitoring processing unit  230  sends a message indicating the number of detected abnormal messages to a first to-be-monitored node  1 , on the Hamilton path, that is operating normally. If none of the to-be-monitored nodes  1  is operating normally, this situation is indicated in the memory  250 . 
         [0060]    Next, processing executed by each to-be-monitored node  1  will be described with reference to  FIG. 11 . 
         [0061]    The communication unit  10  in the to-be-monitored node  1  receives a message including the value of the err_num counter from another apparatus (S 21 ) and outputs the value of the err_num counter to the monitoring processing unit  20 . 
         [0062]    The monitoring processing unit  20  determines whether the local node, to which the monitoring processing unit  20  belongs, is the last node on the Hamilton path (S 23 ). If the monitoring processing unit  20  determines that the local node is the last node, the monitoring processing unit  20  commands the RDMA unit  11  to write the value of the err_num counter to the summary data area  251  in the memory  250  in the management node  200  (S 37 ). Then, the process is terminated. 
         [0063]    If the monitoring processing unit  20  determines that the local node is not the last node, the monitoring processing unit  20  identifies the to-be-monitored node  1  with the Hamilton path&#39;s serial number n, which is stored in the Hamilton path data storage unit  30  (if this step is first executed, the serial number of the local node plus one is set as n). The monitoring processing unit  20  then causes the communication unit  10  to send a message including the value of the err_num counter to the identified to-be-monitored node  1  (S 25 ). 
         [0064]    The monitoring processing unit  20  determines whether the communication unit  10  has received an ACK message from the to-be-monitored node  1  with the serial number n within a predetermined time, that is, a time-out has occurred (S 27 ). If the monitoring processing unit  20  determines that a time-out has not occurred, that is, an ACK message has been received from the to-be-monitored node  1  with the serial number n within the predetermined time, the processing on the to-be-monitored node  1  is terminated. 
         [0065]    If the monitoring processing unit  20  determines that a time-out has occurred, that is, an ACK message has not been received from the to-be-monitored node  1  with the serial number n within the predetermined time, the monitoring processing unit  20  commands the RDMA unit  11  to indicate an error in an area reserved for the serial number n, the area being in the node data area  252  in the memory  250  included in the management node  200 , (S 29 ). For example, the monitoring processing unit  20  turns on a relevant error flag. 
         [0066]    The monitoring processing unit  20  increments the value of the err_num counter by one (S 31 ). The monitoring processing unit  20  then determines whether the n counter has reached the to-be-monitored node count n max  (S 33 ). That is, on the Hamilton path, the monitoring processing unit  20  determines whether a message has been sent from the current to-be-monitored node  1  to all to-be-monitored nodes  1  that are set behind the local node. 
         [0067]    If the monitoring processing unit  20  determines that the n counter has not yet reached the to-be-monitored node count n max , the monitoring processing unit  20  increments the value of the n counter by one (S 35 ). Then, the process returns to S 25 , where the monitoring processing unit  20  sends a message to a next to-be-monitored node  1  on the Hamilton path. 
         [0068]    If the monitoring processing unit  20  determines that the n counter has reached the to-be-monitored node count n max , the process proceeds to S 37 . That is, since there is no normal to-be-monitored node  1  on the Hamilton path behind the local node, the monitoring processing unit  20  notifies the management node  200 . 
         [0069]    In the to-be-monitored node  1 , the RDMA unit  11  writes data directly to the memory  250  in the management node  200  through RDMA. Therefore, a processing load on the management node  200  is not increased. 
         [0070]    Next, processing executed by the management node  200  after the above monitoring processing has been executed will be described with reference to  FIG. 12 . 
         [0071]    First, the abnormal condition processing unit  260  in the management node  200  reads out the number of abnormal nodes (err_num) from the summary data area  251  in the memory  250  (S 41  in  FIG. 12 ). 
         [0072]    The abnormal condition processing unit  260  then determines whether the value of the err_num counter exceeds 0 (S 43 ). If the value of the err_num counter is 0, this indicates that all to-be-monitored nodes  1  are operating normally, so the process is terminated. 
         [0073]    If the abnormal condition processing unit  260  determines that the value of the err_num counter exceeds 0, the abnormal condition processing unit  260  initializes an i counter to 0 (S 45 ). The abnormal condition processing unit  260  then identifies an i-th abnormal node from the node data area  252  (S 47 ). In a case in which a flag is turned on in response to an abnormal condition, the abnormal condition processing unit  260  identifies an abnormal node corresponding to an i-th turned-on flag. 
         [0074]    The abnormal condition processing unit  260  executes processing on the identified abnormal node according to the abnormal condition (S 49 ). Although specific processing is the same as conventional processing and its detailed description will be omitted, the abnormal condition processing unit  260  performs communication again to check the abnormal condition and notifies the manager, for example. 
         [0075]    The abnormal condition processing unit  260  then determines whether the value of the i counter is smaller than the value of the err_num counter (S 51 ). If the abnormal condition processing unit  260  determines that the value of the i counter is smaller than the value of the err_num counter, the abnormal condition processing unit  260  increments the value of the i counter by one (S 53 ). Then, the process proceeds to S 47 . 
         [0076]    If the abnormal condition processing unit  260  determines that the value of the i counter is equal to or larger than the value of the err_num counter, the process is terminated. Then, the management node  200  performs error handling processing. 
         [0077]    When processing is performed as described above, if no error occurs in each to-be-monitored node  1  or each communication link on a Hamilton path, it is possible to carry out monitoring with only a minimum number of communications along a Hamilton path and RDMA communication from the to-be-monitored node  1  at the end point on the Hamilton path to the management node  200 . 
         [0078]    If an error occurs in any one of the to-be-monitored nodes  1 , the to-be-monitored node  1  writes data to the management node  200 . In this case, when RDMA is used, the processing load on the management node  200  can be reduced. 
         [0079]    If an error occurs in a communication path on the Hamilton path, a bypass channel other than the Hamilton path is used for communication. This increase the number of links used. However, the use of a Hamilton path originally minimizes the number of links used. Therefore, the probability of being affected by a link failure is minimized. 
         [0080]    There are a plurality of Hamilton path selection methods. Even after a link failure has been detected, a Hamilton path can be easily set again. 
       Second Embodiment 
       [0081]    In the first embodiment, an example in which only one Hamilton path is set has been described. If many to-be-monitored nodes are present, the Hamilton path may become too long and too much time may be taken in monitoring. 
         [0082]    In this embodiment, to-be-monitored nodes are grouped and a Hamilton path is set for each group so that message transfers are concurrently executed to shorten processing time. 
         [0083]      FIG. 13  outlines a system according to this embodiment. In this embodiment, a management node  200   b  is coupled not only to a node group  100   a  but also to a plurality of other node groups such as a node group  100   b  and a node group  100   c.    
         [0084]      FIG. 14  is a functional block diagram of the management node  200   b  according to this embodiment. The management node  200   b  differs from the management node  200 , illustrated in  FIG. 8 , according to the first embodiment in that a communication unit  240   b  is used instead of the communication unit  240 . The communication unit  240   b  includes a reduction processing unit  242  besides the RDMA unit  241 . In the node data area  252 , areas are allocated for nodes belonging to the node groups  100   a,    100   b,    100   c,  and so on. 
         [0085]    When called from a to-be-monitored node  1 X, the reduction processing unit  242  accumulates a value (err_num) received from the to-be-monitored node  1 X to calculate the total sum of the err_num counter. The reduction processing unit  242  then stores the calculated total sum of the err_num counter in the summary data area  251 . 
         [0086]      FIG. 15  is a functional block diagram of the to-be-monitored node  1 X according to the second embodiment. The to-be-monitored node  1 X in this embodiment differs from the to-be-monitored node  1  according to the first embodiment differs in that a communication unit  10   b  is used instead of the communication unit  10  and the communication unit  10   b  has a reduction processing unit  12  besides the RDMA unit  11 . The reduction processing unit  12  calls the reduction processing unit  242  in the communication unit  240   b  in the management node  200   b  and sends the value of the err_num counter to the reduction processing unit  242 . 
         [0087]    Next, processing executed by the management node  200   b  will be described with reference to  FIGS. 16 and 17 . 
         [0088]    The Hamilton path creating unit  211  in the initialization unit  210  first forms a Hamilton path for each group according to a certain algorithm and stores the Hamilton path in the Hamilton path data storage unit  220  (S 61  in  FIG. 16 ). This processing is the same as in S 1 , except that this processing is executed for each node group. 
         [0089]    The initialization unit  210  causes the communication unit  240   b  to notify each to-be-monitored node  1 X of Hamilton path data including a communication route and a serial number (S 63 ). This processing is the same as in S 3 , except that a different Hamilton path is used for a different node group. 
         [0090]    Then, the monitoring processing unit  230  initializes the err_num counter, which indicates the number of abnormal nodes, to 0. The monitoring processing unit  230  also initializes the n counter, which indicates the node number (serial number) of the to-be-monitored node string on the Hamilton path, to 0. The monitoring processing unit  230  then sets the number of Hamilton paths (that is, the number of groups) in a num_path counter (S 65 ). The monitoring processing unit  230  further initializes the i counter provided for the node group to 0 (S 67 ). 
         [0091]    Next, the monitoring processing unit  230  executes transmission processing for an i-th group (S 69 ). The transmission processing will be described later with reference to  FIG. 17 . 
         [0092]    Upon completion of the transmission processing for the i-th group, the monitoring processing unit  230  increments the value of the i counter by one (S 71 ). The monitoring processing unit  230  then determines whether the value of the i counter is smaller than the value of the num_path counter (S 73 ). If the monitoring processing unit  230  determines that the value of the i counter is smaller than the value of the num_path counter, the monitoring processing unit  230  initializes the n counter to 0 and also initializes the err_num counter to 0 (S 75 ), after which the process returns to S 69 . If the monitoring processing unit  230  determines that the value of the i counter is equal to or larger than the value of the num_path counter, the process is terminated. 
         [0093]    Transmission processing is executed for each node group in this way. 
         [0094]    Next, transmission processing will be described with reference to  FIG. 17 . 
         [0095]    The monitoring processing unit  230  causes the communication unit  240   b  to send a message including the value of the err_num counter to the to-be-monitored node  1 X with the serial number n on the i-th Hamilton path (S 81 ). The communication unit  10   b  in the to-be-monitored node  1 X receives the message and replies with an ACK message. 
         [0096]    The monitoring processing unit  230  determines whether the communication unit  240   b  has received an ACK message from the to-be-monitored node  1 X with the serial number n within a predetermined time, that is, a time-out has occurred (S 83 ). If the monitoring processing unit  230  determines that a time-out has not occurred, that is, an ACK message has been received from the to-be-monitored node  1 X with the serial number n within the predetermined time, the processing by the management node  200   b  at this stage is terminated. 
         [0097]    If the monitoring processing unit  230  determines that a time-out has occurred, that is, an ACK message has not been received from the to-be-monitored node  1 X with the serial number n within the predetermined time, the monitoring processing unit  230  indicates an error in an area, in the node data area  252 , that is reserved for the serial number n (S 85 ). For example, the monitoring processing unit  230  turns on a relevant error flag. 
         [0098]    The monitoring processing unit  230  then increments the err_num counter by one and also increments the n counter by one (S 87 ). The monitoring processing unit  230  determines whether the n counter has reached a to-be-monitored node count n maxi  (S 89 ) on the i-th Hamilton path. That is, the monitoring processing unit  230  determines whether a message has been sent from the management node  200   b  to all to-be-monitored nodes  1 X on the i-th Hamilton path. 
         [0099]    If the monitoring processing unit  230  determines that the n counter has not reached the to-be-monitored node count n maxi , the process returns to S 81 , where the monitoring processing unit  230  sends a message to a next to-be-monitored node  1 X on the Hamilton path. 
         [0100]    If the monitoring processing unit  230  determines that the n counter has reached the to-be-monitored node count n maxi , this indicates that none of the to-be-monitored nodes  1 X is operating normally. In this case, the monitoring processing unit  230  adds the value of the err_num counter, which indicates the number of abnormal nodes, the value being the total to-be-monitored node count n maxi , to a value stored in the summary data area  251  (S 91 ). It is assumed that 0 has been stored in the summary data area  251  as an initial value. Then, the process returns to the processing on the calling node. 
         [0101]    A message including the value of the err_num counter can be sent for each Hamilton path in this way. If there is no to-be-monitored node  1 X that is operating normally, a result indicating this situation is accumulated in the summary data area  251 . 
         [0102]    Each to-be-monitored node  1 X executes processing illustrated in  FIG. 18 . 
         [0103]    The communication unit  10   b  in the to-be-monitored node  1 X first receives a message including the value of the err_num counter from another apparatus (S 101  in  FIG. 18 ) and outputs the value of the err_num counter to the monitoring processing unit  20 . 
         [0104]    The monitoring processing unit  20  determines whether the local node, to which the monitoring processing unit  20  belongs, is the last node on the Hamilton path (S 103 ). If the monitoring processing unit  20  determines that the local node is the last node, the monitoring processing unit  20  causes the reduction processing unit  12  in the communication unit  10   b  to call the reduction processing unit  242  in the management node  200   b  and output the value of the err_num counter to the reduction processing unit  242  (S 117 ). Then, the process is terminated. The reduction function is used in this way to ask the reduction processing unit  242  in the management node  200   b  to sum the value of the err_num counter. 
         [0105]    If the monitoring processing unit  20  determines that the local node is not the last node, the monitoring processing unit  20  identifies the to-be-monitored node  1 X with the Hamilton path&#39;s serial number n, which is stored in the Hamilton path data storage unit  30  (if this step is first executed, the serial number of the local node plus one is set as n). The monitoring processing unit  20  then causes the communication unit  10   b  to send a message including the value of the err_num counter to the identified to-be-monitored node  1 X (S 105 ). 
         [0106]    The monitoring processing unit  20  determines whether the communication unit  10   b  has received an ACK message from the to-be-monitored node  1 X with the serial number n within a predetermined time, that is, a time-out has occurred (S 107 ). If the monitoring processing unit  20  determines that a time-out has not occurred, that is, an ACK message has been received from the to-be-monitored node  1 X with the serial number n within the predetermined time, the processing on the to-be-monitored node  1 X is terminated. 
         [0107]    If the monitoring processing unit  20  determines that a time-out has occurred, that is, an ACK message has not been received from the to-be-monitored node  1 X with the serial number n within the predetermined time, the monitoring processing unit  20  commands the RDMA unit  11  to indicate an error in an area reserved for the serial number n, the area being in the node data area  252  in the memory  250  included in the management node  200   b,  (S 109 ). For example, the monitoring processing unit  20  turns on a relevant error flag. Alternatively, for example, if the local node belongs to a node group with a number in the range from 10,000 to 19,999, the monitoring processing unit  20  calculates a serial number in the entire system by adding n to 10,000 and indicates an error in an area reserved for the node with the serial number in the entire system. 
         [0108]    The monitoring processing unit  20  increments the value of the err_num counter by one (S 111 ). The monitoring processing unit  20  then determines whether the n counter has reached the to-be-monitored node count n max  on the Hamilton path to which the local node belongs (S 113 ). That is, the monitoring processing unit  20  determines whether a message has been sent from the current to-be-monitored node  1 X to all to-be-monitored nodes  1 X on the Hamilton path that are set behind the local node. 
         [0109]    If the monitoring processing unit  20  determines that the n counter has not yet reached the to-be-monitored node count n max , the monitoring processing unit  20  increments the value of the n counter by one (S 115 ). Then, the process returns to S 105 , where the monitoring processing unit  20  sends a message to a next to-be-monitored node X 1  on the Hamilton path. 
         [0110]    If the monitoring processing unit  20  determines that the n counter has reached the to-be-monitored node count n max , the process proceeds to S 117 . That is, since there is no normal to-be-monitored node  1 X behind the local node on the Hamilton path, the monitoring processing unit  20  notifies the management node  200   b.    
         [0111]    In the to-be-monitored node  1 X, the RDMA unit  11  writes data directly to the memory  250  in the management node  200   b  through RDMA. Therefore, a processing load on the management node  200   b  is not increased. 
         [0112]    The to-be-monitored node  1 X at the end of the to-be-monitored nodes  1 X, on the Hamilton path, that operate normally outputs the value of the err_num counter to the reduction processing unit  242  in the management node  200   b  in S 117 . 
         [0113]    Then, the reduction processing unit  242  in the management node  200   b  executes processing as illustrated in  FIG. 19 . 
         [0114]    Specifically, the reduction processing unit  242  is called from the to-be-monitored node  1 X and receives the value of the err_num counter (S 121 ). The reduction processing unit  242  then updates the total of the err_num counter by adding the received value of the err_num counter to the value stored in the summary data area  251  and writing the resulting value to the summary data area  251  (S 123 ). 
         [0115]    When this processing is executed, even if the value of the err_num counter is asynchronously received from the to-be-monitored node  1 X that is essentially at the end of each Hamilton path, the value stored in the summary data area  251  is updated as processing by the communication unit  240   b.  Therefore, a processing load on the central processing unit (CPU) in the management node  200   b  can be suppressed. 
         [0116]    Although both the first and second embodiments have been described by assuming an N-dimensional mesh, the effects described above can be obtained by executing similar processing on an N-dimensional torus. In the case of a torus, the to-be-monitored node at the start point on a Hamilton path and the to-be-monitored node at the end point may be the same. However, it suffices for the node to operate as the node at the start point if a node from which a message is sent is a management node, and to operate as the node at the end point if a node from which a message is sent is another to-be-monitored node. 
         [0117]    This completes the descriptions of the embodiments of the present technology. The present technology is not limited to these embodiments. For example, the functional blocks described above may not necessarily match the structure of parts of the apparatus or its program module structure. 
         [0118]    Although examples of sending and receiving a message including the value of the err_num counter have been described, other data may be included in a message. For example, a bit string corresponding to to-be-monitored nodes present on the route on a Hamilton path may be included in a message. If a to-be-monitored node to which to send the message is found to be abnormal, the bit corresponding to the abnormal to-be-monitored node to which to send the message may be turned on. 
         [0119]    Furthermore, the order in which a processing flow is executed may be changed if the processing result remains unchanged. Alternatively, a plurality of steps may be concurrently executed. 
         [0120]    The management nodes  200  and  200   b  and the to-be-monitored nodes  1  and  1 X are each a computer apparatus. As illustrated in  FIG. 20 , the computer apparatus includes a memory  2501 , a CPU  2503 , a hard disk drive (HDD)  2505 , a display controller  2507  coupled to a display apparatus  2509 , a drive unit  2513  for a removable disk  2511 , an input unit  2515 , and a communication controller  2517  coupled to a network; these components are mutually coupled through a bus  2519 . An OS and application programs that execute processing in the embodiments of the present technology are stored in the HDD  2505 . When these programs are executed by the CPU  2503 , they are read out from the HDD  2505  and stored in the memory  2501 . The CPU  2503  controls the display controller  2507 , communication controller  2517 , and drive unit  2513  according to the processing executed by the application program so that the application program operates as predetermined. Data during processing is stored mainly in the memory  2501 . However, the data may be stored in the HDD  2505 . In the embodiments of the present technology, the application programs that execute the processing described above are distributed in the form of being stored on the removable disk  2511 , which is computer-readable, and are installed from the drive unit  2513  in the HDD  2505 . In some cases, the application programs are installed in the HDD  2505  through a network such as the Internet and the communication controller  2517 . In this type of computer apparatus, hardware components such as the CPU  2503  and memory  2501  described above and programs such as the OS and application programs efficiently cooperate to implement the functions described above. 
         [0121]    The communication controller  2517  corresponds to communication unit  240  in the management node  200  or to the communication unit  240   b  in the management node  200   b.  Alternatively, the communication controller  2517  corresponds to the communication unit  10  in the to-be-monitored node  1  or to the communication unit  10   b  in the to-be-monitored node  1 X. That is, the communication controller  2517  has an RDMA function and a reduction function. 
         [0122]    The embodiments described above will be summarized below. 
         [0123]    The systems according to the embodiments each include a plurality of information processing apparatuses that are coupled through a communication path and a management apparatus that manages the plurality of information processing apparatuses. The management apparatus sends a message, which includes information about a detected abnormal condition, to a first information processing apparatus that is operating normally, the first information processing apparatus being one of the plurality of information processing apparatuses on a Hamilton path, which is set for the plurality of information processing apparatuses. When each of the plurality of information processing apparatuses receives a message, if a next information processing apparatus that is operating normally is present on the Hamilton path, the information processing apparatus sends a message including information about abnormal conditions that have been detected so far to the next information processing apparatus that is operating normally. If a next information processing apparatus that is operating normally is unable to be detected on the Hamilton path or the information processing apparatuses that has received the message is the information processing apparatus at the end of the Hamilton path, the information processing apparatus sends information about abnormal conditions that have been detected so far to the management apparatus. 
         [0124]    Thus, a processing load to detect an abnormal condition can be reduced in the entire system. Specifically, in principle, only one message transmission and reception is sufficient at each information processing apparatus. At the management apparatus, in principle, the number of message transmissions is suppressed to one on each Hamilton path, and the number of receptions of information about a detected abnormal condition is also suppressed to one on each Hamilton path. In transmission of information about a detected abnormal condition to the management apparatus, a message may be used or RDMA may be used. 
         [0125]    Each of the plurality of information processing apparatuses searches for a next information processing apparatus, on the Hamilton path, that is operating normally. If the information processing apparatus detects an abnormal information processing apparatus, the information processing apparatus may write data about the abnormal information processing apparatus to a memory in the management apparatus. When RDMA is used in this way, a processing load on the processor in the management apparatus can be reduced. 
         [0126]    The management apparatus described above may send a message including information about a detected abnormal condition to a first normally operating information processing apparatus of a plurality of information processing apparatuses on a Hamilton path that is set for one of a plurality of groups to which one of the plurality of information processing apparatuses belongs. If the management apparatus receives the information about abnormal conditions that have been detected so far for each of the plurality of groups, the management apparatus may summarizes the information. If a Hamilton path is set for each group of information processing apparatuses, processing is concurrently executed in a case in which many information processing apparatuses are handled, so processing time can be shortened. 
         [0127]    Programs that cause a computer to execute processing as described above can be created. These programs are stored, for example, on a computer-readable recording medium or in a storage unit; examples of the computer-readable recording medium include a flexible disk, optical disks such as a compact disk-read-only memory (CD-ROM), a magneto-optic disk, a semiconductor memory (such as, for example, a ROM), and a hard disk. Data being processed is temporarily stored in a storage unit such as a random-access memory (RAM). 
         [0128]    All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.