Abstract:
A determination method, for determining a possibility of a new failure in a system, includes: obtaining first setting values for a plurality of setting items of the system when a failure in the system occurs; obtaining second setting values for the plurality of setting items when an input that the failure has been recovered is received; identifying at least one setting item from among the plurality of setting items based on the first setting values and the second setting values, the at least one setting item having a first setting value different from a second setting value; determining a value from among the first value and the second value of the at least one setting item; comparing an input value regarding the at least one setting item and the value; determining the possibility based on a result of the comparing; and outputting information regarding the possibility.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2014-201501, filed on Sep. 30, 2014, the entire contents of which are incorporated herein by reference. 
       FIELD 
       [0002]    The embodiments discussed herein are related to a technique for determining whether there have been signs of failure occurrence or not. 
       BACKGROUND 
       [0003]    To date, services have been provided in which a system operated by a user side is monitored, and if a sign of failure occurrence in the system is detected, a notification is sent to a system administrator, or the like of the user side that the sign of failure occurrence has been detected. 
         [0004]    For example, as a method of monitoring processes, a proposal is made of a method including a step of generating a signature representing a process, a step of continuously updating the generated signature, and a step of detecting abnormality based on the continuously updated signature. Related-art technique is disclosed in Japanese Laid-open Patent Publication No. 2004-348740, for example. 
         [0005]    Also, for a system including at least one processor, and one software application, a proposal is made of a system of monitoring collected diagnostic data in order to automatically make a diagnosis on a wireless device by a processor. In this system, problem patterns that have occurred up to now are learned in order to improve the diagnostic ability on all the problems. Related-art technique is disclosed in Japanese Laid-open Patent Publication No. 2007-052756, for example. 
       SUMMARY 
       [0006]    According to an aspect of the invention, a determination method executed by a computer for determining a possibility of a new failure in a system, the determination method includes: obtaining first setting values set for a plurality of setting items of the system when a failure in the system occurs; obtaining second setting values set for the plurality of setting items when an input that the failure has been recovered is received; identifying at least one setting item from among the plurality of setting items based on the first setting values and the second setting values, the at least one setting item having a first setting value different from a second setting value; determining a value from among the first value and the second value of the at least one setting item, the value being used for the determining of the possibility; comparing an input value regarding the at least one setting item and the value; determining the possibility based on a result of the comparing; and outputting information regarding the possibility. 
         [0007]    The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
         [0008]    It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0009]      FIG. 1  is a schematic diagram illustrating a system configuration including a sign detection apparatus according to the present embodiment; 
           [0010]      FIG. 2  is a functional block diagram of the sign detection apparatus according to the present embodiment; 
           [0011]      FIG. 3  is an explanatory diagram illustrating a relationship between the occurrence of a failure and recovery, and collection of configuration information; 
           [0012]      FIG. 4  is a diagram illustrating an example of case data; 
           [0013]      FIG. 5  is a diagram illustrating an example of a failure type list; 
           [0014]      FIG. 6  is a diagram illustrating an example of a key list; 
           [0015]      FIG. 7  is a diagram illustrating an example of a pattern list; 
           [0016]      FIG. 8  is a diagram illustrating an example of a disregard list; 
           [0017]      FIG. 9  is a diagram illustrating an example of a learning data database; 
           [0018]      FIG. 10  is a diagram illustrating an example of a count data database; 
           [0019]      FIG. 11  is a diagram illustrating an example of a specific score database; 
           [0020]      FIG. 12  is a diagram illustrating an example of a sign detection result list; 
           [0021]      FIG. 13  is a hardware configuration diagram of a computer that functions as the sign detection apparatus according to the present embodiment; 
           [0022]      FIG. 14  is a flowchart illustrating an example of learning processing; 
           [0023]      FIG. 15  is a flowchart illustrating an example of pattern generation processing; 
           [0024]      FIG. 16  is a flowchart illustrating an example of learning data generation processing; 
           [0025]      FIG. 17  is a flowchart illustrating an example of specific score calculation processing; and 
           [0026]      FIG. 18  is a flowchart illustrating an example of detection processing. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
       [0027]    One of the causes of the occurrence of a failure in a system is an error in setting values for various setting items in the configuration information that defines a system configuration. Further, a setting error derives from the case of setting an erroneous value, and the case of omission of changing a value that has to be changed. When a failure occurs in a system, it is possible for a user to assume that a setting value is erroneous. However, if omission of changing a value is the cause of failure occurrence, it is difficult for the user to estimate the cause of the failure, and to determine detection of a sign of failure occurrence. 
         [0028]    With related-art techniques, detection and learning of a failure are performed using information before and after the occurrence of a failure. However, in this case, it is difficult to detect a failure occurrence that is caused by omission of changing a value. 
         [0029]    Also, when a failure occurs in a system, and its recovery work is carried out, the configuration information before the failure recovery and the configuration information after the failure recovery are compared so that a setting item of the configuration information related to the failure is identified. Thus, the configuration information is stored for a certain period, and learning is performed with a setting item value that was changed before and after the failure recovery as information on the setting change item. Then, it is thought that a sign of failure occurrence is detected based on the comparison between a value set in the setting items of newly input configuration information, and the setting change item information. However, it is difficult to identify which one of the values before and after the failure recovery is to be used as suitable setting change item information depending on a failure type and a difference in the setting items, and the like. 
         [0030]    According to an embodiment of the disclosed technique, it is desirable to identify information on setting change items suitable for detecting a sign of failure occurrence. 
         [0031]    In the following, a detailed description will be given of an example of an embodiment according to the present disclosure with reference to the drawings. 
         [0032]    As illustrated in  FIG. 1 , a sign detection apparatus  10  according to the present embodiment is connected to a processing system  14  including a plurality of processing apparatuses  16  through a network  12 , such as the Internet, or the like. The sign detection apparatus  10  monitors each of the processing apparatuses  16 , and detects a sign of failure occurrence in each of the processing apparatuses  16 . 
         [0033]    It is possible to detect a sign of failure occurrence in one of the processing apparatuses  16  by determining whether correct values are set or not for the setting values of the various setting items in the configuration information (the details will be described later) in the processing apparatus  16 , for example. For a method of determining whether the set value is correct or not, an assumption is made of a method of comparing the set value with correct learning data, or erroneous learning data, which are provided in advance. When the set value and a value of the correct learning data are compared, it is possible to determine that the set value is correct if the two values match. When the set value and a value of the erroneous learning data are compared, it is possible that the set value is incorrect if the two values match. 
         [0034]    However, there are two causes of failure occurrence in the processing apparatus  16 . One of the causes is the case of setting erroneous data for each one of the setting items of the configuration information, and the other of the causes is the case of omission of changing the value that has to be changed. In this manner, there are differences in the causes of failure occurrence, and thus it is difficult to determine which one of correct learning data or erroneous learning data is to be compared with the set value in order to make it possible to carry out more suitable sign detection. 
         [0035]    For example, in the case where a setting item value having a predetermined initial value has to be changed to be a suitable value for each of the processing apparatuses  16 , if there is omission of changing the value, the setting item value remains the initial value, and thus this becomes a cause of failure occurrence. In this case, there is a possibility that a correct setting value is different for each of the processing apparatuses  16 , and thus it is not easy to prepare correct learning data that cover all of these values. In this case, learning data that indicates an error if the initial value is set ought to be provided, and it is possible to more suitably detect a sign of failure occurrence by comparing the erroneous learning data and the set value. 
         [0036]    Thus, in the sign detection apparatus  10  according to the present embodiment, both correct learning data, and erroneous learning data for comparing with the set value are provided, and learning data that allows more suitable detection of a sign of failure occurrence is identified and used. 
         [0037]    Also, when a sign of failure occurrence is detected by comparing the values of before and after a configuration information change, it is not possible to detect a case where omission of change of setting values causes failure occurrence. This is because there is no change if the values of before and after a configuration information change are compared. In the present embodiment, correct learning data and erroneous learning data are generated from the configuration information of before and after a failure recovery, and a comparison is made between the learning data and the set values so that the set value is determined to be correct or not. 
         [0038]      FIG. 2  illustrates a functional block diagram of the sign detection apparatus  10 . As illustrated in  FIG. 2 , the sign detection apparatus  10  includes a learning unit  20  including a pattern generation unit  21 , a learning data generation unit  22 , and a specific score calculation unit  23 , and a detection unit  24 . Also, a storage unit  30  stores a learning data database (DB)  31 , a count data DB  32 , and a specific score DB  33 . Descriptions will been given later of the individual databases. In this regard, the learning unit  20  is an example of a sign detection support apparatus according to the present disclosure. The pattern generation unit  21  is an example of a storage unit and an extraction unit according to the present disclosure. Also, the learning data generation unit  22  and the specific score calculation unit  23  are examples of specification units according to the present disclosure. 
         [0039]    Here, as illustrated in  FIG. 3 , when a failure occurs in the processing apparatus  16 , configuration information is collected immediately after the failure occurrence in order to confirm the state at the time of the failure occurrence. Then, after the failure is recovered, the configuration information is collected in order to record the recovery work and confirm the operation, and the like. A pair of the configuration information that is collected before and after the failure recovery is input into the sign detection apparatus  10  as case data. In this regard, the configuration information is information indicating the hardware configuration of the processing apparatus  16 , and the software configuration of the operating system (OS), the applications, and the like that are installed in the processing apparatus  16 . The configuration information is, for example data of the directory structure, which is the information extracted on the configuration from the file system of the processing apparatus  16  using a known dedicated tool. 
         [0040]      FIG. 4  illustrates an example of case data  34 . In the example in  FIG. 4 , configuration information  35 A collected before a failure recovery is recorded together with its collection time. Also, configuration information  35 B collected after the failure recovery is recorded together with its collection time. Further, a failure type  36 , which is identification information determined for each failure type, is recorded. The failure type  36  is recorded in the case data  34  by an operator, or the like who collects the configuration information at the time of failure recovery, for example. 
         [0041]    The pattern generation unit  21  receives a plurality of case data  34  as input, and stores the data into a predetermined storage area. Also, the pattern generation unit  21  records the failure types  36  included in the plurality of the stored case data  34 , respectively in a failure type list  37  as illustrated in  FIG. 5 , for example. In this regard, a duplicate failure type is not recorded. 
         [0042]    Also, the pattern generation unit  21  extracts all the keys identifying various setting items on the configuration from each of the configuration information  35 A before the failure recovery, and the configuration information  35 B after the failure recovery, which are included in the individual case data  34 . For example, as described above, in the configuration information of a directory structure, a key is expressed by a path from a root directory to a file, and a parameter name to be set in the file. Thus, the pattern generation unit  21  extracts “/etc/my.cnf:port” as a key from the description in the first row “/etc/my.cnf:port=3306” of the configuration information  35 A before the failure recovery in the case data  34  in  FIG. 4 , for example. The pattern generation unit  21  lists the extracted keys, and creates a key list  38  as illustrated in  FIG. 6 , for example. 
         [0043]    Also, the pattern generation unit  21  generates a pattern in which a failure type, a key, and values before and after the failure recovery are associated if the values before and after the failure recovery are different for each key recorded in the key list  38 . For example, in the case data in  FIG. 4 , for the key “/etc/my.cnf:port”, the value before the failure recovery “3306” and the value after the failure recovery “3309” are different. Accordingly, the pattern, in which the failure type “F001”, the key “/etc/my.cnf:port”, the value before the failure recovery “3306”, and the value after the failure recovery “3309” are associated, is generated. The pattern generation unit  21  records the pattern generated for each key into a pattern list  39  as illustrated in  FIG. 7 , for example. In the example in  FIG. 7 , “value V A ” is the value before the failure recovery, and “value V B   ”  is the value after the failure recovery. 
         [0044]    Further, the pattern generation unit  21  deletes a pattern having the failure type and the key that match a pair of a failure type and a key predetermined in a disregard list  40  as illustrated in  FIG. 8 , for example from the pattern list  39 . In the disregard list  40 , a key by which it is difficult to find a cause and a sign of failure occurrence by comparing the values before and after the failure recovery, such as a key that changes every time a system is started, or the like, for example is predetermined for each failure type. 
         [0045]    The learning data generation unit  22  generates learning data from each pattern recorded in the pattern list  39  generated by the pattern generation unit  21 . The learning data is summary data for a certain key, which includes the number of occurrences of a certain value as a correct answer, and the number of occurrences of a certain value as an error for each failure type. The pattern recorded in the pattern list  39  includes the values before and after the failure recovery for each key. The value before the failure recovery V A  is an erroneous value, and the value after the failure recovery V B  is a correct value. 
         [0046]    For example, as illustrated in  FIG. 9 , a plurality of learning data including items of a failure type, a key, a success or failure, a value, and the number of times are recorded in the learning data DB  31 . The learning data generation unit  22  increases, by one, the number of times of the learning data having a failure type, a key, and a value that match the failure type, the key, and the value V A  before the failure recovery of a certain pattern, respectively, and having a success or failure of “Failure” for one pattern. Also, the learning data generation unit  22  increases, by one, the number of times of the learning data having a failure type, a key, and a value that match the failure type, the key, and the value V B  after the failure recovery of a certain pattern, respectively, and having a success or failure of “Success” for one pattern. In this regard, if the learning data having a failure type, a key, and a value that match the failure type, the key, and the value V A  before the failure recovery or the value V B  after the failure recovery is not recorded in the learning data DB  31 , the learning data generation unit  22  adds the relevant learning data, and sets the number of times to 1. 
         [0047]    Also, for example as illustrated in  FIG. 10 , the learning data generation unit  22  records the number of times N S , which is the number produced by counting the number of learning data having the success or failure of “Success” for each failure type and for each key, into the count data DB  32 . In the same manner, the learning data generation unit  22  records the number of times N F , which is the number produced by counting the number of learning data having the success or failure of “Failure” for each failure type and for each key, into the count data DB  32 . 
         [0048]    The specific score calculation unit  23  calculates a specific score for identifying whether to use the learning data having a correct value, or an erroneous value out of the learning data when a sign of failure occurrence is detected from the newly input configuration information. The specific score represents that the higher the probability of having a same value as a value for a certain key, that is to say, the fewer the number of variations of a value for the certain key, the more probable the value is a correct answer, or an error. 
         [0049]    For example, the specific score calculation unit  23  obtains the empirical probability of the occurrence of each value of the learning data having the success or failure of “Success” for a certain key for a certain failure type in the learning data DB  31 . Then, the specific score calculation unit  23  calculates the conditional entropy from the obtained probability, and determines the conditional entropy to be a specific score S S  of a correct answer indicating the probability of the learning data having the success or failure of “Success”. In the same manner, the specific score calculation unit  23  calculates the conditional entropy from the empirical probability of the occurrence of each value of the learning data having the success or failure of “Failure”, and determines the conditional entropy to be a specific score S F  of a correct answer indicating the probability of the learning data having the success or failure of “Failure”. The specific score S S  is expressed by the following expression (1), and the specific score S F  is expressed by the following expression (2). In this regard, X Success  is a set of the learning data having the success or failure of “Success” for a certain key for a certain failure type, and X Failure  is a set of the learning data having the success or failure of “Failure”. 
         [0000]    
       
         
           
             
               
                 
                   
                     S 
                     S 
                   
                    
                   
                     = 
                     def 
                   
                    
                   
                     
                       ∑ 
                       
                         x 
                         ∈ 
                         
                           X 
                           Success 
                         
                       
                     
                      
                     
                         
                     
                      
                     
                       
                         - 
                         
                           p 
                            
                           
                             ( 
                             
                               x 
                                
                               Success 
                             
                             ) 
                           
                         
                       
                        
                       log 
                        
                       
                           
                       
                        
                       
                         P 
                          
                         
                           ( 
                           
                             x 
                              
                             Success 
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
             
               
                 
                   
                     S 
                     F 
                   
                    
                   
                     = 
                     def 
                   
                    
                   
                     
                       ∑ 
                       
                         x 
                         ∈ 
                         
                           X 
                           Failure 
                         
                       
                     
                      
                     
                         
                     
                      
                     
                       
                         - 
                         
                           p 
                            
                           
                             ( 
                             
                               x 
                                
                               Failure 
                             
                             ) 
                           
                         
                       
                        
                       log 
                        
                       
                           
                       
                        
                       
                         P 
                          
                         
                           ( 
                           
                             x 
                              
                             Failure 
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
         [0050]    More specifically, a description will be given of an example of calculating the specific score S S  and the specific score S F  for the failure type “F001” and the key “/etc/my.cnf:port” using the learning data DB  31  illustrated in  FIG. 9 , and the count data DB  32  illustrated in  FIG. 10 . In this case, X Success  and X Failure  are as follows. 
         [0051]    X Success ={3309} 
         [0052]    X Failure ={3306, 3307, 3308} 
         [0053]    In this regard, the individual learning data included in the above-described sets represent values included in the learning data. 
         [0054]    The specific score calculation unit  23  obtains the number of occurrences (three times) of the learning data of the value “3309” included in the XSuccess from the learning data DB  31 . In the same manner, the specific score calculation unit  23  obtains the number of occurrences (each one time) of each learning data of the value “3306”, the value “3307”, and the value “3308” that are included in X Failure  from the learning data DB  31 . Also, the specific score calculation unit  23  obtains the number of occurrences N S  (three times) of the learning data having the failure type “F001”, the key “/etc/my.cnf:port”, and the success or failure of “Success”, and the number of occurrences N F  (three times) of the learning data having the success or failure of “Failure” from the count data DB  32 . 
         [0055]    The specific score calculation unit  23  calculates an empirical probability for each value of the learning data using the obtained number of times as illustrated below. 
         [0056]    p(3306|Failure)=1/3 
         [0057]    p(3307|Failure)=1/3 
         [0058]    p(3308|Failure)=1/3 
         [0059]    p(3309|Success)=3/3 
         [0060]    The specific score calculation unit  23  calculates the specific score S S  and the specific score S F  as follows using the calculated empirical probability by the above-described expressions (1) and (2). 
         [0000]        S   S =−3/3 log 3/3=0,  S   F =−3×3/3 log 3/3=0.48
 
         [0061]    The specific score calculation unit  23  calculates the specific score S S  and the specific score S F  for each failure type and for each key, and stores the scores in the specific score DB  33  as illustrated in  FIG. 11 , for example. 
         [0062]    When the configuration information to be a target of sign detection is input, the detection unit  24  detects a sign of failure occurrence using the learning data DB  31 , the count data DB  32 , and the specific score DB  33  stored in the storage unit  30 . Specifically, the detection unit  24  compares each detection target data represented by pairs of the key and the value included in the detection-target configuration information, and the learning data, and determines whether the value of each setting item in the configuration information is correctly set or not. If determined that the correct value is not set, the detection unit  24  detects a sign of failure occurrence, and outputs the sign detection result. 
         [0063]    As described above, in the present embodiment, after identification of using either correct learning data or erroneous learning data is performed, sign detection is carried out. Specifically, the detection unit  24  obtains the specific score S S  and the specific score S F  corresponding to the key that matches a key included in the detection target data for each failure type from the specific score DB  33 . The smaller the value of the specific score S S  illustrated in the above-described expression (1), the higher the probability that the value of the correct learning data is a correct answer is indicated. Also, the smaller the value of the specific score S F  illustrated in the above-described expression (2), the higher the probability that the value of the erroneous learning data is an error is indicated. Thus, for a failure type having the specific score S S  smaller than the specific score S F , the detection unit  24  identifies the correct learning data, and for a failure type having the specific score S F  smaller than the specific score S S , the detection unit  24  identifies the erroneous learning data. 
         [0064]    For a failure type for which the correct learning data is identified, the detection unit  24  compares the detection target data and the correct learning data, and if they do not match, the detection unit  24  detects a sign of failure occurrence. Also, for a failure type for which erroneous learning data is identified, the detection unit  24  compares the detection target data and the erroneous learning data, and if they match, the detection unit  24  detects a sign of failure occurrence. If the detection unit  24  has detected a sign of failure occurrence, the detection unit  24  records a sign detection result in which a detection score (the details will be described later) is given to the failure type, and the detection target data (key and value) in a sign detection result list  41  as illustrated in  FIG. 12 , for example. 
         [0065]    The detection score is a score indicating the probability of the sign detection result. For example, it is assumed that there are a plurality of erroneous learning data having the same key as the key of the detection target data for a failure type, and the value of the detection target data matches any one of the erroneous learning data. In this case, the higher is the number of occurrences of the erroneous learning data that have matched the value of the detection target data, the higher the probability that the value is an error becomes. Thus, it is possible for the detection unit  24  to determine a value produced by dividing the number of occurrences N of the erroneous learning data that has matched the value of the detection target data by the number of occurrences N F  of the erroneous learning data having the same failure type and key, for example to be a detection score. In this regard, it is possible to obtain the number of occurrences N from the learning data DB  31 , and to obtain the number of occurrences N F  from the count data DB  32 . 
         [0066]    Also, if the value of the correct learning data having the same key as the key of the detection target data does not match the value of the detection target data for a certain failure type, it is not possible to calculate a detection score based on the number of occurrences as described above. Thus, the detection unit  24  gives a value (for example, “−1”) indicating that there has been no match with the correct learning data as the detection score unlike the detection score based on the number of occurrences described above. 
         [0067]    It is possible to achieve the sign detection apparatus  10  by a computer  50  illustrated in  FIG. 13 , for example. The computer  50  includes a CPU  51 , a memory  52  as a temporary storage area, and a nonvolatile storage device  53 . Also, the computer  50  includes an input-output interface (I/F)  54  to which an input-output device  58  is connected. Also, the computer  50  includes a read/write (R/W) unit  55 , which controls reading and writing data from and to a recording medium  59 , and a network I/F  56  through which a network  12 , such as the Internet, or the like is connected. The CPU  51 , the memory  52 , the storage device  53 , the input-output I/F  54 , the R/W unit  55 , and the network I/F  56  are mutually connected through a bus  57 . 
         [0068]    It is possible to achieve the storage device  53  by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage device  53 , as a storage medium, stores a sign detection program  60  for causing the computer  50  to function as a sign detection apparatus  10 . Also, the storage device  53  includes a learning data storage area  71  for storing learning data, a count data storage area  72  for storing count data, and a specific score storage area  73  for storing a specific score. 
         [0069]    The CPU  51  reads the sign detection program  60  from the storage device  53 , loads the program into the memory  52 , and executes the processes of the sign detection program  60  in sequence. Also, the CPU  51  reads the learning data stored in the learning data storage area  71 , and loads the data into the memory  52  as the learning data DB  31 . Also, the CPU  51  reads the count data stored in the count data storage area  72 , and loads the data into the memory  52  as the count data DB  32 . Also, the CPU  51  reads the specific score stored in the specific score storage area  73 , and loads the score into the memory  52  as the specific score DB  33 . Further, the CPU  51  creates the above-described failure type list  37 , key list  38 , pattern list  39 , and sign detection result list  41  in the memory  52  during the execution of the sign detection program  60 . 
         [0070]    The sign detection program  60  includes a pattern generation process  61 , a learning data generation process  62 , a specific score calculation process  63 , and a detection process  64 . 
         [0071]    The CPU  51  executes the pattern generation process  61  so as to operate as the pattern generation unit  21  illustrated in  FIG. 2 . Also, the CPU  51  executes the learning data generation process  62  so as to operate as the learning data generation unit  22  illustrated in  FIG. 2 . Also, the CPU  51  executes the specific score calculation process  63  so as to operate as the specific score calculation unit  23  illustrated in  FIG. 2 . Also, the CPU  51  executes the detection process  64  so as to operate as the detection unit  24  illustrated in  FIG. 2 . Thereby, the computer  50  executing the sign detection program  60  functions as the sign detection apparatus  10 . 
         [0072]    In this regard, it is possible to achieve the sign detection apparatus  10  by a semiconductor integrated circuit, more specifically by an application specific integrated circuit (ASIC), or the like, for example. 
         [0073]    Next, a description will be given of operation of the sign detection apparatus  10  according to the present embodiment. First, when a plurality of case data  34  is input into the sign detection apparatus  10 , learning processing illustrated in  FIG. 14  is executed in the sign detection apparatus  10 , and the learning data DB  31 , the count data DB  32 , and the specific score DB  33  are stored in the storage unit  30 . In this state, when detection-target configuration information is input into the sign detection apparatus  10 , detection processing illustrated in  FIG. 18  is executed in the sign detection apparatus  10 , and the sign detection result list  41  is output. In this regard, the learning processing and the detection processing that are executed by the sign detection apparatus  10  exemplify a sign detection method according to the present disclosure. In the following, a detailed description will be given of each processing. 
         [0074]    In step S 10  of the learning processing illustrated in  FIG. 14 , the pattern generation unit  21  executes pattern generation processing the details of which are illustrated in  FIG. 15 . 
         [0075]    In step S 11  of the pattern generation processing illustrated in  FIG. 15 , the pattern generation unit  21  obtains one piece of the case data  34  from a plurality of input case data  34 . The case data  34  includes the configuration information  35 A before the failure recovery, the configuration information  35 B after the failure recovery, and the failure type  36 . Also, the pattern generation unit  21  obtains the disregard list  40  stored in a predetermined area (omitted in illustration). 
         [0076]    Next, in step S 12 , the pattern generation unit  21  records the failure type  36  illustrated in the obtained case data  34  into the failure type list  37  as illustrated in  FIG. 5 , for example. 
         [0077]    Next, in step S 13 , the pattern generation unit  21  extracts all the keys from the configuration information  35 A before the failure recovery, and the configuration information  35 B after the failure recovery, which are included in the obtained case data  34 , and creates the key list  38  as illustrated in  FIG. 6 , for example. 
         [0078]    Next, in step S 14 , the pattern generation unit  21  determines whether there are keys for which the processing in step S 15  to S 17  has not been performed in the key list  38  or not. If there are unprocessed keys, the processing proceeds to step S 15 , the pattern generation unit  21  selects one from the unprocessed keys, and determines it to be a key K. 
         [0079]    Next, in step S 16 , the pattern generation unit  21  obtains the values before and after the failure recovery corresponding to the key K from the configuration information  35 A before the failure recovery, and the configuration information  35 B after the failure recovery, respectively. Then, the pattern generation unit  21  determines whether the obtained values before and after the failure recovery are different or not. If they are different, the processing proceeds to step S 17 , and if they are equal, the processing returns to step S 14 . 
         [0080]    In step S 17 , the pattern generation unit  21  generates a pattern in which the failure type included in the case data  34  obtained in step S 11 , the key K, and the value before the failure recovery V A , and the value after the failure recovery V B , which are corresponding to the key K, are associated. Then, the pattern generation unit  21  adds the generated pattern to the pattern list  39  as illustrated in  FIG. 7 , for example. 
         [0081]    If determined that there are no unprocessed keys in step S 14  described above, the processing proceeds to step S 18 . In step S 18 , the pattern generation unit  21  deletes a pattern including a failure type and a key that match the predetermined pair of the failure type and the key in the disregard list  40  obtained in step S 11  from the pattern list  39 . 
         [0082]    Next, in step S 19 , the pattern generation unit  21  outputs the generated failure type list  37 , and pattern list  39 . If the pattern generation processing for all the input case data  34  is completed, the processing returns to the learning processing illustrated in  FIG. 14 . 
         [0083]    Next, in step S 20  of the learning processing illustrated in  FIG. 14 , the learning data generation unit  22  executes the learning data generation processing the details of which are illustrated in  FIG. 16 . 
         [0084]    In step S 21  of the learning data generation processing illustrated in  FIG. 16 , the learning data generation unit  22  obtains the pattern list  39  output from the pattern generation unit  21 . 
         [0085]    Next, in step S 22 , the learning data generation unit  22  determines whether there are patterns for which the following processing in step S 23  to S 26  has not been performed in the pattern list  39  or not. If there are unprocessed patterns, the processing proceeds to step S 23 . In step S 23 , the learning data generation unit  22  selects one pattern from the unprocessed patterns, and determines that the failure type included in the pattern to be F, the key to be K, the value before the failure recovery to be V A , and the value after the failure recovery to be V B . 
         [0086]    Next, in step S 24 , the learning data generation unit  22  increases the number of times of the learning data having the failure type of F, the key of K, the value of V A , and the success or failure of “Failure” by one in the learning data DB  31  as illustrated in  FIG. 9 , for example. Next, in step S 25 , the learning data generation unit  22  increases the number of times of the learning data having the failure type of F, the key of K, the value of V B , and the success or failure of “Success” by one in the learning data DB  31 . In this regard, in step S 24  and S 25 , if the relevant learning data is not recorded in the learning data DB  31 , the learning data generation unit  22  adds the relevant learning data, and then sets the number of times to 1. 
         [0087]    Next, in step S 26 , the learning data generation unit  22  increases the count data (N S  and N F ) having the failure type of F, and the key of K by one, respectively in the count data DB  32  as illustrated in  FIG. 10 , for example, and the processing returns to step S 22 . In this regard, if the relevant count data is not recorded in the count data DB  32 , the learning data generation unit  22  adds the relevant count data, and then individually sets N S  and N F  to 1. 
         [0088]    In step S 22 , if determined that there are no unprocessed, the processing proceeds to step S 27 , the learning data generation unit  22  outputs the learning data DB  31  and the count data DB  32 , and the processing returns to the learning processing illustrated in  FIG. 14 . 
         [0089]    Next, in step S 30  of the learning processing illustrated in  FIG. 14 , the specific score calculation unit  23  executes the specific score calculation processing the details of which are illustrated in  FIG. 17 . 
         [0090]    In step S 31  of the specific score calculation processing illustrated in  FIG. 17 , the specific score calculation unit  23  obtains the failure type list  37  output from the pattern generation unit  21 , and the learning data DB  31  and the count data DB  32 , which are output from the learning data generation unit  22 . 
         [0091]    Next, in step S 32 , the specific score calculation unit  23  determines whether there are failure types for which the following processing in step S 33  to S 40  has not been performed in the failure type list  37 . If there are unprocessed failure types, the processing proceeds to step S 33 , and the specific score calculation unit  23  selects one failure type from the unprocessed failure types, and determines it to be the failure type F. 
         [0092]    Next, in step S 34 , the specific score calculation unit  23  extracts all the keys included in the learning data having the failure type of F out of the learning data recorded in the learning data DB  31 , and creates the key list of F. Next, in step S 35 , the specific score calculation unit  23  determines whether there are keys for which the following processing in step S 36  to S 40  has not been performed in the key list of F. If there are unprocessed keys, the processing proceeds to step S 36 , the specific score calculation unit  23  selects one key from the unprocessed keys, and determines the key to be K. 
         [0093]    Next, in step S 37 , the specific score calculation unit  23  obtains the count data (N S  and N F ) having the failure type of F, and the key of K from the count data DB  32 . 
         [0094]    Next, in step S 38 , the specific score calculation unit  23  obtains the number of times of the learning data having the failure type of F, the key of K, and the success or failure of “Success” for each value of the learning data from the learning data DB  31 . Then, the specific score calculation unit  23  obtains an empirical probability for each value using the number of times N S  obtained in step S 37 , and the number of times obtained from the learning data, and calculates the specific score S S  of the correct answer by the expression (1), for example. 
         [0095]    Next, in step S 39 , the specific score calculation unit  23  obtains the number of times of the learning data having the failure type of F, the key of K, and the success or failure of “Failure” for each value of the learning data from the learning data DB  31 . Then, the specific score calculation unit  23  obtains an empirical probability for each value using the number of times N F  obtained in step S 37 , and the number of times obtained from the learning data, and calculates the specific score S F  of the error by the expression (2), for example. 
         [0096]    Next, in step S 40 , the specific score calculation unit  23  records a group of the failure type of F, the key of K, the specific score of S S , and the specific score of S F  in the specific score DB  33  as illustrated in  FIG. 11 , for example, and the processing returns to step S 35 . 
         [0097]    In step S 35 , if determined that there are no unprocessed keys, the processing returns to step S 32 . In step S 32 , if determined that there are no unprocessed failure types, the processing proceeds to step S 41 . 
         [0098]    In step S 41 , the specific score calculation unit  23  stores the learning data DB  31 , the count data DB  32 , and the specific score DB  33  into the storage unit  30 , then the processing returns to  FIG. 14 , and the learning processing is terminated. 
         [0099]    Next, in step S 51  of the detection processing illustrated in  FIG. 18 , the detection unit  24  obtains the input detection-target configuration information. Also, the detection unit  24  obtains the learning data DB  31 , the count data DB  32 , and the specific score DB  33  stored in the storage unit  30 . 
         [0100]    Next, in step S 52 , the detection unit  24  determines whether there are detection target data for which the processing of the following step S 53  to S 63  has not been performed out of the detection target data represented by a pair of the key and the value included in the detection-target configuration information. If there are unprocessed detection target data, the processing proceeds to step S 53 , the detection unit  24  selects one of the unprocessed detection target data, and determines the key to be K included in the selected detection target data, and the value to be V. 
         [0101]    Next, in step S 54 , the detection unit  24  determines whether there are failure types for which the processing of the following step S 55  to S 63  has been performed out of the failure types recorded in the specific score DB  33  correspondingly to the key K. If there are unprocessed failure types, the processing proceeds to step S 55 , the detection unit  24  selects one failure type from the unprocessed failure types to determine it to be F, and obtains the specific score S S  and the specific score S F  corresponding to the failure type of F, and the key of K from the specific score DB  33 . 
         [0102]    Next, in step S 56 , the detection unit  24  compares the specific score S S  and the specific score S F , and determines if S S &lt;S F  or not. As described above, the lower the value of the specific score S S  expressed in the (1) expression, the higher the probability that the value of the correct learning data is the correct answer is represented. Also, the lower the value of the specific score S F  expressed in the (2) expression, the higher the probability that the value of the erroneous learning data is an error is represented. Accordingly, if S S &lt;S F , it is possible to conduct sign detection more suitably using the correct learning data, whereas if S S &gt;S F , it is possible to conduct sign detection more suitably using the erroneous learning data. If S S =S F , either learning data may be used. However, in the present embodiment, it is assumed to use the erroneous learning data. If S S &lt;S F , the processing proceeds to step S 57 , and if S S  S F , the processing proceeds to step S 60 . 
         [0103]    In step S 57 , the detection unit  24  obtains the value of the learning data having the failure type of F, the key of K, and the success or failure of “Success” from the learning data DB  31 , and determines the value to be V R . Next, in step S 58 , the detection unit  24  determines whether the value V of the detection target data matches the value V R  of the obtained learning data. If V=V R , it indicates that the correct value is set in the value V of the detection target data, and thus the processing directly returns to step S 54 . If V≠V R , the processing proceeds to step S 59 , and the detection unit  24  sets a value “−1” in the detection score S, which indicates that the value of the detection target data has not matched the correct learning data, and the processing proceeds to step S 63 . 
         [0104]    On the other hand, in step S 60 , the detection unit  24  obtains the value of the learning data having the failure type of F, the key of K, and the success or failure of “Failure” from the learning data DB  31 , and determines the value to be V R . Next, in step S 61 , the detection unit  24  determines whether the value V of the detection target data matches the value V R  of the obtained learning data. If V≠V R , an erroneous value is not said to be set in the value V of the detection target data, and thus the processing directly returns to step S 54 . If V=V R , the processing proceeds to step S 62 . In step S 62 , the detection unit  24  obtains the number of occurrences N of the erroneous learning data having the failure type of F, the key of K, and the value of V R  from the learning data DB  31 . Also, the detection unit  24  obtains the number of occurrences N F  of the erroneous learning data having the failure type of F, and the key of K from the count data DB  32 . Then, the detection unit  24  sets the detection score S to N/N F , and the processing proceeds to step S 63 . 
         [0105]    In step S 63 , the detection unit  24  records a group of the failure type of F, the key of K, the value of V, and the detection score of S into the sign detection result list  41  as illustrated in  FIG. 12  as a result of the sign detection, for example, and the processing returns to step S 54 . In step S 54 , if determined that there are no unprocessed failure types, the processing returns to step S 52 . In step S 52 , if determined that there are no unprocessed detection target data, the processing proceeds to step S 64 . 
         [0106]    In step S 64 , the detection unit  24  outputs the sign detection result list  41 , and the detection processing is terminated. 
         [0107]    In this regard, if a plurality of values V R  are obtained in step S 57 , and if determined that the value V of the detection target data does not match any one of the values V R  in step S 58 , the processing ought to proceed to step S 59 . 
         [0108]    Also, if a plurality of values V R  are obtained in step S 60 , and if determined that the value V of the detection target data matches any one of the values V R  in step S 61 , the processing ought to proceed to step S 62 . Also, the threshold value S th  for the detection score S may be set in advance, and only the sign detection result having a relationship S&gt;S th  may be added to the sign detection result list  41 . Thereby, it is possible to include only the case of having a high probability of a sign of failure occurrence in the sign detection result list  41 . Also, a different value for each failure type and for each key may be set to the threshold value S th . 
         [0109]    As described above, with the present embodiment, learning data is generated from each piece of the configuration information obtained before and after a failure recovery, a comparison is made between the learning data, and the detection target data included in the detection-target configuration information, and a determination is made whether a correct value is set in each setting item or not. Thereby, it is possible to detect a sign of failure occurrence caused by omission of changing a setting value, which is not possible to be detected by a method of detecting a part having different values between before and after a change in the configuration information, and analyzing whether that part might cause the failure. 
         [0110]    Also, with the present embodiment, learning data having a higher probability as a correct answer, or an error is used out of correct learning data and erroneous learning data. Thereby, it is possible to suitably detect a sign of failure occurrence caused by both an error in changing a setting value, and omission of changing a setting value. 
         [0111]    In this regard, in the above-described embodiments, it is assumed that each time a failure occurs in the processing apparatus  16 , the configuration information is collected both before and after the recovery of the failure. Accordingly, it is assumed that the pieces of the configuration information  35 A and  35 B included in the input case data  34  are identified as the configuration information of before and after the failure recovery. 
         [0112]    However, the configuration information is sometimes collected at periodic timing (for example, once a day), or is sometimes collected at any timing regardless of a failure recovery. If the configuration information collected at such timing is input, the configuration information whose collection time is included in a predetermined period as a period before the failure recovery is identified as the configuration information  35 A before the failure recovery. In the same manner, the configuration information whose collection time is included in a predetermined period as a period after the failure recovery is identified as the configuration information  35 B after the failure recovery. Then, the configuration information  35 A before the failure recovery, and the configuration information  35 B after the failure recovery, which were collected in the predetermined periods corresponding to the periods before and after the failure recovery, respectively ought to be formed into a pair, and a pattern ought to be generated for a setting item whose value has been changed. 
         [0113]    For example, the configuration information that is collected during operation other than failure recovery operation sometimes includes a setting item in which a value that is not originally a correct answer, or a value unrelated with the failure recovery is set. If learning data is generated from such configuration information, unsuitable learning data is generated as a correct answer or erroneous learning data. Accordingly, the “predetermined period” for identifying periods before and after the failure recovery is determined to be a period excluding an operation period unrelated with the failure recovery operation. 
         [0114]    Also, in the above-described embodiments, a description has been given of the case including a failure type, a key, a value, and a detection score in a sign detection result. However, a correct value may be added to the sign detection result. Specifically, in the case where sign detection is conducted using correct learning data, it is possible to add a value of correct learning data as a correct value. Also, in the case where sign detection is conducted using erroneous learning data, it is possible to add a value of the learning data having the same failure type and key as those of the learning data, and the success or failure of “Success” as a correct value. 
         [0115]    Also, in the above, a description has been given of the mode in which the sign detection program  60 , which is an example of a sign detection program according to the present disclosure, is stored (installed) in the storage device  53 . However, the present disclosure is not limited to this. It is possible to provide a sign detection program according to the disclosed technique in a form of the sign detection program recorded in a recording medium, such as a CD-ROM, a DVD-ROM, a USB memory, or the like. 
         [0116]    All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.