Abstract:
Disclosed is a technique wherein an object that requires adjustment in order to increase the reliability of automatic classification can be easily identified. A device ( 140 ) for adjusting classification classifies defects into a first class group according to the feature amount of the defects that are obtained from image data obtained from an electron microscope ( 110 ), and classifies the defects into a second class group according to the feature amount of the defects classified into the first class group. And, the device ( 140 ) for adjusting the classification calculates classification performance by comparing the defects that have been classified into the second class group, and outputs the calculated classification performance in a predetermined display format to an output unit ( 180 ).

Description:
TECHNICAL FIELD 
       [0001]    The present invention relates to a technology of classifying defects into class groups by attribute data of the defects obtained from image data, which is obtained by imaging a test object. The present invention claims priority from Japanese Patent Application No. 2009-172104 filed on Jul. 23, 2009, and the content therein is incorporated herein by reference for designated countries that allow incorporation of documents by reference. 
       BACKGROUND ART 
       [0002]    In recent years, for the purposes of quick recognition of the state of occurrence of defects on a surface of a wafer represented by a semiconductor integrated circuit and monitoring of the number of the occurred defects for each type of the defects, there have been developed technologies of taking an image of a defect portion for automatic classification. 
         [0003]    One of the technologies of performing the automatic classification from image data of the defect portion is a technology called learning classification. In the learning classification technology, image data for learning is collected in advance and learned, to thereby optimize a classification model. Representative methods of the learning classification technology include discriminant analysis based on a neural network and the Bayes discriminant theory and the like. 
         [0004]    Another technology of the automatic classification from the image data of the defect portion is a technology called a rule-based classification. In the rule-based classification technology, attribute data is extracted from image data, and a value of the attribute data is determined from an “IF-THEN” rule incorporated in a system, to thereby classify defects. 
         [0005]    Further, in Patent Literature 1, a method of improving classification performance by combining a plurality of classification models is described. In the method described in Patent Literature 1, a classification model formed of a hierarchy of multiple levels of rough defect classification called main classes and detailed defect classification called sub classes is used. 
       CITATION LIST 
     Patent Literature 
       [0000]    
       
         Patent Literature 1: JP 2004-294360 A 
       
     
       SUMMARY OF INVENTION 
     Technical Problem 
       [0007]    With the technology described in Patent Literature 1, the disclosed classification model formed of the hierarchy of the multiple levels can classify defects with performance higher than the defect classification by the learning classification or the rule-based classification. 
         [0008]    However, even with the technology described in Patent Literature 1, when purity, which is one of indices for measuring classification performance, is lower than a fixed value, for example, a defect classification result, which is an output of the automatic classification, cannot be trusted, and visual classification by a person is needed. The term “purity” as used herein can be defined as a ratio of defects that should be truly classified as a class, of defects automatically classified as the class. 
         [0009]    In view of the above-mentioned circumstances, the present invention has an object of providing a technology capable of easily identifying objects that should be adjusted for increasing reliability of the automatic classification. 
       Solution to Problem 
       [0010]    In order to attain the above-mentioned object, in the present invention, the classification performance of the defects classified as the class is evaluated to be output in a predetermined format. 
         [0011]    For example, according to the present invention, there is provided a defect classification apparatus which classifies, based on attribute data of defects obtained from image data obtained by taking an image of a test object, the defects into a group of first classes, and classifies, based on the attribute data of the defects classified into the group of first classes, the defects into a group of second classes, including: a storage unit which stores, for each of the defects, classification answer information which identifies the group of second classes to which the each of the defects is to be classified; and a control unit, the control unit performing the processings of: classifying the defects into the group of second classes based on a value obtained by multiplying a likelihood value determined for each of the group of first classes, each of the group of second classes, and each type of the attribute data by a value of a weight coefficient determined for the each of the group of first classes, the each of the group of second classes, and the each type of the attribute data; comparing the second classes to which the defects are classified and the second class to which the defects are to be classified, which is identified by the classification answer information, to calculate classification performance in each combination of the first classes and the second classes and in the second classes to which the defects are classified from all the first classes; and outputting classification performance information which associates the calculated classification performance with the each combination of the first classes and the second classes, and the second classes to which the defects are classified from all the first classes, to an output unit. 
       Advantageous Effects of Invention 
       [0012]    As described above, according to the present invention, the objects that should be adjusted for increasing the reliability of the automatic classification may be easily identified. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0013]      FIG. 1  is a schematic diagram of a defect classification apparatus. 
           [0014]      FIG. 2  is a schematic diagram of a classification recipe adjustment device. 
           [0015]      FIG. 3  is a schematic diagram of a defect information table. 
           [0016]      FIG. 4  is a schematic diagram of a weight coefficient information table. 
           [0017]      FIG. 5  is a schematic diagram of a function for identifying a value of attribute data and a likelihood value. 
           [0018]      FIG. 6  is a schematic diagram of a classification performance information table. 
           [0019]      FIG. 7  is a schematic diagram illustrating an example of a hierarchical classification model. 
           [0020]      FIG. 8  is a schematic diagram of a comparison table. 
           [0021]      FIG. 9  is a schematic diagram of a comparison table showing classification performance of a sub classification class A. 
           [0022]      FIG. 10  is a schematic diagram of a comparison table showing classification performance of a sub classification class B. 
           [0023]      FIG. 11  is a schematic diagram of a comparison table showing classification performance of a sub classification class C. 
           [0024]      FIG. 12  is a schematic diagram of a comparison table showing classification performance of a sub classification class D. 
           [0025]      FIG. 13  is a schematic diagram of a computer. 
           [0026]      FIG. 14  is a flow chart illustrating extraction processing by an extraction unit and processing of adjusting a sub classification model by an adjustment unit. 
           [0027]      FIG. 15  is a schematic diagram of an orthogonal table. 
           [0028]      FIG. 16  is a schematic diagram of a reevaluation table in a main class. 
           [0029]      FIG. 17  is a schematic diagram of a reevaluation table in total. 
           [0030]      FIG. 18  is a schematic diagram of a first factor effect diagram. 
           [0031]      FIG. 19  is a schematic diagram of a second factor effect diagram. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
       [0032]      FIG. 1  is a schematic diagram of a defect classification apparatus  100  according to an embodiment of the present invention. As illustrated in the figure, the defect classification apparatus  100  includes a scanning electron microscope (SEM)  110 , a general control unit  130 , a classification recipe adjustment device  140 , a secondary storage device  160 , an input unit  170 , and an output unit  180 , which are connected to one another via a bus  190 . 
         [0033]    The SEM  110  includes an electron source  111  for generating primary electrons  118 , an acceleration electrode  112  for accelerating the primary electrons, a focusing lens  113  for converging the primary electrons  118 , a deflector  114  for two-dimensionally scanning and deflecting the primary electrons  118 , an objective lens  115  for converging the primary electrons  118  on a sample  116 , a drive stage  117  on which the sample  116  is mounted, a detector  120  for detecting a secondary electron signal  119  generated from the sample  116 , a digital conversion unit  121  for converting the signal detected by the detector  120  into a digital signal, and reflection electron detectors  123   a  and  123   b  for detecting a reflection electron signal  122 , and those parts are connected to the general control unit  130  via the bus  190 . 
         [0034]    In  FIG. 1 , two reflection electron detectors  123   a  and  123   b  are arranged to be opposed to each other, and detect different components of the reflection electron signal  122  emitted from the sample  116 . 
         [0035]    The general control unit  130  controls overall processing in the defect classification apparatus  100 . 
         [0036]    The classification recipe adjustment device  140  performs automatic defect classification processing for automatically classifying defects from image data of an image taken by the SEM  110 , and adjustment processing for adjusting a classification model in the automatic defect classification processing. Note that, the classification recipe adjustment device  140  is described in detail with reference to  FIG. 2 . 
         [0037]    The secondary storage device  160  stores various parameters such as imaging conditions of the SEM  110  and operation conditions of the drive stage  117 . 
         [0038]    The input unit  170  receives input of information. 
         [0039]    The output unit  180  outputs information. 
         [0040]      FIG. 2  is a schematic diagram of the classification recipe adjustment device  140 . As illustrated in the figure, the classification recipe adjustment device  140  includes a storage unit  141 , a control unit  148 , and an interface unit  153 . 
         [0041]    The storage unit  141  includes a defect information storage region  142 , a weight coefficient information storage region  143 , a likelihood information storage region  144 , a classification performance information storage region  145 , and an adjustment information storage region  146 . 
         [0042]    The defect information storage region  142  stores defect information for identifying attribute data of the defects extracted from the image data of the image taken by the SEM  110 . For example, in this embodiment, a defect information table  142   a  as illustrated in  FIG. 3  (schematic diagram of the defect information table  142   a ) is stored. 
         [0043]    As illustrated in the figure, the defect information table  142   a  includes a defect number column  142   b , an attribute data X 1  column  142   c , an attribute data X 2  column  142   d , an attribute data X 3  column  142   e , an attribute data X 4  column  142   f , an attribute data X 5  column  142   g , an attribute data X 6  column  142   h , an attribute data X 7  column  142   i , and an attribute data X 8  column  142   j.    
         [0044]    The defect number column  142   b  stores a defect number, which is identification information for identifying a defect extracted from the image data of the image taken by the SEM  110 . 
         [0045]    The attribute data X 1  column  142   c  stores a value corresponding to predetermined attribute data X 1  of the defect identified in the defect number column  142   b.    
         [0046]    The attribute data X 2  column  142   d  stores a value corresponding to predetermined attribute data X 2  of the defect identified in the defect number column  142   b.    
         [0047]    The attribute data X 3  column  142   e  stores a value corresponding to predetermined attribute data X 3  of the defect identified in the defect number column  142   b.    
         [0048]    The attribute data X 4  column  142   f  stores a value corresponding to predetermined attribute data X 4  of the defect identified in the defect number column  142   b.    
         [0049]    The attribute data X 5  column  142   g  stores a value corresponding to predetermined attribute data X 5  of the defect identified in the defect number column  142   b.    
         [0050]    The attribute data X 6  column  142   h  stores a value corresponding to predetermined attribute data X 6  of the defect identified in the defect number column  142   b.    
         [0051]    The attribute data X 7  column  142   i  stores a value corresponding to predetermined attribute data X 7  of the defect identified in the defect number column  142   b.    
         [0052]    The attribute data X 8  column  142   j  stores a value corresponding to predetermined attribute data X 8  of the defect identified in the defect number column  142   b.    
         [0053]    Note that, in this embodiment, the attribute data of the defects is specified as eight types X 1  to X 8 , but the number of types of the attribute data is not limited to eight. 
         [0054]    Returning to  FIG. 2 , the weight coefficient information storage region  143  stores, for each combination of a sub class of the classification model and the attribute data of the defect, weight coefficient information for identifying a weight coefficient value used for classifying sub classes. For example, in this embodiment, a weight coefficient information table  143   a  as illustrated in  FIG. 4  (schematic diagram of the weight coefficient information table  143   a ) is stored for each main class. Note that,  FIG. 4  is the coefficient information table  143   a  of a main class B, but a coefficient information table similar to that of  FIG. 4  is stored for each main class. 
         [0055]    As illustrated in the figure, the weight coefficient information table  143   a  includes an attribute data identification row  143   b , a sub class identification row  143   c , and a value identification region  143   d.    
         [0056]    The attribute data identification row  143   b  stores information for identifying the attribute data of the defect. For example, in this embodiment, values of “k” in a weight coefficient W B,s,k  (in this case, “1” to “8”) correspond to the attribute data X 1  to X 8 . 
         [0057]    The sub class identification row  143   c  stores information for identifying the sub class of the classification model. For example, in this embodiment, values of “s” in the weight coefficient W B,s,k  (in this case, “1” to “5”) correspond to “sub class  1 ” to “sub class  5 ”. 
         [0058]    The value identification region  143   d  stores, for each main class, in a region identified by the attribute data identification row  143   b  and the sub class identification row  143   c , a weight coefficient value corresponding to the attribute data identified by the attribute data identification row  143   b  and a sub class identified by the sub class identification row  143   c.    
         [0059]    The likelihood information storage region  144  stores likelihood information for identifying a value of the attribute data and a likelihood value corresponding to the value of the attribute data. For example, in this embodiment, a function for identifying the value of the attribute data and the likelihood value as illustrated in  FIG. 5  (schematic diagram of the function for identifying the value of the attribute data and the likelihood value) is stored in advance for each combination of the main classes (in this case, “A” to “D”), the sub classes (in this case, “1” to “5”), and the attribute data (in this case, “1” to “8”). 
         [0060]    Returning to  FIG. 2 , the classification performance information storage region  145  stores classification performance information for identifying a value of classification performance for each combination of a main class and sub classes classified from the main class and a value of the classification performance for each sub class. For example, in this embodiment, a classification performance information table  145   a  as illustrated in  FIG. 6  (schematic diagram of the classification performance information table  145   a ) is stored. 
         [0061]    As illustrated in the figure, the classification performance information table  145   a  includes a sub class column  145   b , a main class A column  145   c , a main class B column  145   d , a main class C column  145   e , a main class D column  145   f , and an aggregate column  145   g.    
         [0062]    The sub class column  145   b  stores information for identifying sub classes. Here, in this embodiment, as the information for identifying the sub classes, information for identifying a name of each sub class, and information for identifying a type of defects classified as each sub class are stored. 
         [0063]    The main class A column  145   c  stores information for identifying the classification performance when defects are classified from a main class A as the sub class identified by the sub class column  145   b . Here, as the information for identifying the classification performance in this embodiment, the purity (%), an accuracy (%), and the number of defects for which the result of visual classification and the result of automatic classification are the same are stored. Note that, details of the purity and the accuracy are described later. 
         [0064]    The main class B column  145   d  stores information for identifying the classification performance when defects are classified from a main class B as the sub class identified by the sub class column  145   b.    
         [0065]    The main class C column  145   e  stores information for identifying the classification performance when defects are classified from a main class C as the sub class identified by the sub class column  145   b.    
         [0066]    The main class D column  145   f  stores information for identifying the classification performance when defects are classified from a main class D as the sub class identified by the sub class column  145   b.    
         [0067]    The aggregate column  145   g  stores information for identifying the classification performance when defects are classified as the sub class identified by the sub class column  145   b.    
         [0068]    Returning to  FIG. 2 , the adjustment information storage region  146  stores adjustment information for identifying the classification performance of the sub class when the weight coefficient value is changed. Note that, the adjustment information stored in the adjustment information storage region  146  is described later. 
         [0069]    The control unit  148  includes a classification unit  149 , an extraction unit  150 , and an adjustment unit  151 . 
         [0070]    The classification unit  149  classifies defects identified by the defect information stored in the defect information storage region  142  as a main class by a predetermined algorithm, and further classifies the defects classified as the main class into sub classes by using an algorithm that is different from the predetermined algorithm used to classify the defects as the main class. 
         [0071]    For example, when the classification unit  149  acquires image data of a defect from the SEM  110  as illustrated in  FIG. 7  (schematic diagram illustrating an example of hierarchical classification model), the classification unit  149  extracts a plurality of pieces of attribute data from the acquired image data and stores the pieces of extracted attribute data in the defect information table  142   a.    
         [0072]    Then, the classification unit  149  inputs those pieces of attribute data to a main classification model. The main classification model classifies the defects based on the input attribute data. In the example of  FIG. 7 , a large number of defects are classified into four types of main classes A, B, C, and D. 
         [0073]    Next, the classification unit  149  inputs the attribute data to a sub classification model that is modeled for each of the main classes, and classifies the defects into detailed sub classes. In this example, defects classified as the main class A by the main classification model are classified into the sub class  2  and the sub class  5  by a sub classification model A, defects classified as the main class B are classified into the sub class  1 , the sub class  2 , and the sub class  5  by a sub classification model B, defects classified as the main class C are classified into the sub class  1  and the sub class  2  by a sub classification model C, and defects classified as the main class D are classified into the sub class  3  and the sub class  4  by a sub classification model D. 
         [0074]    In such hierarchical classification model, the main classification model has the role of rough classification, and hence various methods such as an IF-THEN rule-based classification model and a learning classification model based on the Bayes discriminant theory may be applied. On the other hand, the sub classification model has the role of detailed classification, and hence the learning classification model is suitable. Note that, the algorithm used by the main classification model and the algorithm used by the sub classification model are different. 
         [0075]    For example, in this embodiment, the classification unit  149  uses the mathematical expression represented by Equation (1) below as the sub classification model with respect to defects classified as the same main class, and classifies the defects as a sub class in which each defect has the largest value of y. 
         [0000]    
       
         
           
             
               
                 
                   
                     [ 
                     
                       Equation 
                        
                       
                           
                       
                        
                       1 
                     
                     ] 
                   
                    
                   
                       
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     y 
                     
                       m 
                       , 
                       s 
                       , 
                       d 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         k 
                         - 
                         1 
                       
                       N 
                     
                      
                     
                       ( 
                       
                         
                           W 
                           
                             m 
                             , 
                             s 
                             , 
                             k 
                           
                         
                          
                         
                           
                             P 
                             
                               m 
                               , 
                               s 
                               , 
                               k 
                             
                           
                            
                           
                             ( 
                             
                               x 
                               
                                 k 
                                 , 
                                 d 
                               
                             
                             ) 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0076]    Here, w denotes the weight coefficient, and P(x) denotes the likelihood. Of the subscript indices of y, w, and P, m represents a main class number assigned to each main class to identify the main class (that is, sub classification model number), s represents a sub class number assigned to each sub class to identify the sub class, k represents a number allocated to each piece of the attribute data to identify the attribute data, d represents the defect number allocated to each defect to identify the defect, and y is calculated for each of the main class number, the sub class number, and the defect number. N is the number of types of the attribute data, which is, in this example, N=8. 
         [0077]    Then, to W m,s,k , the weight coefficient corresponding to the sub class number and the attribute data number is extracted from the weight coefficient information table  143   a  stored for each main class in the weight coefficient information storage region  143  and substituted. 
         [0078]    Further, the value of a likelihood P(x) is acquired for the value x of the attribute data from the function of the likelihood stored in the likelihood information storage region  144 . Note that, the function of the likelihood is defined for each combination of the main class number, the sub class number, and the attribute data number as described above. Note that, the function (distribution) of the likelihood is obtained by aggregating the learning data when the classification model is learned. 
         [0079]    Further, the classification unit  149  receives via the input unit  170 , for each defect identified by the defect information stored in the defect information storage region  142 , an input of a visual classification result obtained by visual (manual) inspection and classification into the sub classes by the user, and aggregates the purity (%), the accuracy (%), and the number of defects for which the result of visual classification and the result of automatic classification are the same to be stored in the classification performance information table  145   a.    
         [0080]    Note that, the visual inspection result includes information for identifying at least the defect number, the sub class to which the defect identified by the defect number is classified, and when the input is received, the classification unit  149  stores the input as classification answer information in the storage unit  141 . 
         [0081]    Further, the classification unit  149  may also generate a comparison table  191   a  as illustrated in  FIG. 8  (schematic diagram of the comparison table  191   a ) to be output to the output unit  180 . 
         [0082]    As illustrated in the figures, comparison table  191   a  includes Class  1  column  191   b , Class  2  column  191   c , Class  3  column  191   d , Class  4  column  191   e , Class  5  column  191   f , Subtotal column  191   g , Accuracy column  191   h , Class  1  row  191   j , Class  2  row  191   k , Class  3  row  191   l , Class  4  row  191   m , Class  5  row  191   n , Subtotal row  191   o , and Purity row  191   p.    
         [0083]    Further, in the cell that intersect each of Class  1  column  191   b , Class  2  column  191   c , Class  3  column  191   d , Class  4  column  191   e , and Class  5  column  191   f , and Class  1  row  191   j , Class  2  row  191   k , Class  3  row  191   l , Class  4  row  191   m , and Class  5  row  191   n , the aggregated values of the results of visual classification and automatic classification are stored. 
         [0084]    For example, in each of the Class  1  column  191   b , the Class  2  column  191   c , the Class  3  column  191   d , the Class  4  column  191   e , and the Class  5  column  191   f , the number of defects classified as each of the sub classes  1  to  5  in the automatic classification by the classification unit  149  is stored, and in each of the Class  1  row  191   j , the Class  2  row  191   k , the Class  3  row  191   l , the Class  4  row  191   m , and the Class  5  row  191   n , the number of defects classified as each of the sub classes  1  to  5  by the visual classification, of the defects classified as each of the sub classes  1  to  5  in the automatic classification by the classification unit  149 . 
         [0085]    A numerical value in each of the cells of the sub classes for which the row and the column are the same, that is, on the diagonal line from the upper left to the lower right of the comparison table  191   a , is the number of defects for which the results of visual classification and automatic classification are the same and indicates the number of defects for which the automatic classification is correctly performed. On the other hand, a numerical value in each of the cells deviated from the diagonal line indicates the number of defects for which the results of visual classification and automatic classification are different. 
         [0086]    Further, in Subtotal column  191   g , the aggregated value (added value) of values stored in each row of Class  1  row  191   j , Class  2  row  191   k , Class  3  row  191   l , Class  4  row  191   m , and Class  5  row  191   n  is stored, and in Accuracy column  191   h , the accuracy value of the calculated value stored in each row of Class  1  row  191   j , Class  2  row  191   k , Class  3  row  191   l , Class  4  row  191   m , and Class  5  row  191   n  is stored. 
         [0087]    Further, in Subtotal row  191   o , the aggregated value (added value) of values stored in each column of Class  1  column  191   b , Class  2  column  191   c , Class  3  column  191   d , Class  4  column  191   e , and Class  5  column  191   f  is stored, and in Purity row  191   p , the purity value calculated by the value stored in each column of Class  1  column  191   b , Class  2  column  191   c , Class  3  column  191   d , Class  4  column  191   e , and Class  5  column  191   f  is stored. 
         [0088]    Note that, the accuracy as a value indicating the classification performance is a value obtained by dividing, for each row of the table, the numerical value on the diagonal line by a subtotal in the second column from the right and multiplying the resultant by “100”. On the other hand, the purity as a value indicating the classification performance is a value obtained by dividing, for each column of the table, the numerical value on the diagonal line by a subtotal in the second row from the bottom and multiplying the resultant by “100”. 
         [0089]      FIGS. 9 to 12  are comparison tables  192   a  to  195   a  obtained by dividing the comparison table  191   a  illustrated in  FIG. 8  by the main classes.  FIG. 9  is the comparison table  192   a  representing the classification performance of a sub classification class A, FIG.  10  is the comparison table  193   a  representing the classification performance of a sub classification class B,  FIG. 11  is the comparison table  194   a  representing the classification performance of a sub classification class C, and  FIG. 12  is the comparison table  195   a  representing the classification performance of a sub classification class D. 
         [0090]    The classification unit  149  may also output those comparison tables  192   a  to  195   a  to the output unit  180 . 
         [0091]    The extraction unit  150  identifies, in the aggregate column  145   g  of the classification performance information table  145   a  illustrated in  FIG. 6 , a cell for which the value of the purity is less than 75% and the number of defects for which the result of visual classification and the result of automatic classification are the same is 10 or more, and displays the identified cell in a manner that is distinguishable from other cells (for example, in a color that is different from the other cells). 
         [0092]    Similarly, the extraction unit  150  identifies, in the aggregate column  145   g  of the classification performance information table  145   a  illustrated in  FIG. 6 , and in rows including cells for which the value of the purity is less than 75% and the number of defects for which the result of visual classification and the result of automatic classification are the same is 10 or more, from among the main classes in which the value of the purity is stored, a main class for which the value of the purity is less than 75% and the number of defects for which the result of visual classification and the result of automatic classification are the same is 10 or more, and displays cells included in the identified main class (cells included in the column of the identified main class) in a manner that is distinguishable from other cells (for example, in a color that is different from the other cells). 
         [0093]    Similarly, the extraction unit  150  identifies, from among the cells included in the main class identified as described above, a cell for which the number of defects for which the result of visual classification and the result of automatic classification are the same is 10 or more, and displays the identified cell in a manner that is distinguishable from other cells (for example, in a color that is different from the other cells). Further, the extraction unit  150  identifies the sub class corresponding to the thus identified cell from the sub class column  145   b , and identifies the identified sub class as the sub class to be adjusted. 
         [0094]    The above-mentioned processing by the extraction unit  150  is based on the following analysis. 
         [0095]    For example, the user of the defect classification apparatus  100  may first check the rightmost column of the classification performance information table  145   a  illustrated in  FIG. 6  to recognize that the purities of the sub class  4  and the sub class  5  are less than 75%. 
         [0096]    Then, with respect to the sub class  4 , the number of defects for which the result of visual classification and the result of automatic classification are the same is as small as 3, and hence it can be recognized that, even when the purity of the sub class  4  is improved, the sub class  4  does not make much contribution to reducing the number required for visual classification by a person. On the other hand, with respect to the sub class  5 , the number of defects for which the result of visual classification and the result of automatic classification are the same is 43, and hence it can be recognized that, if it is possible to improve the purity of the sub class  5 , the result of automatic classification is reliable and the number required for visual classification by a person can be reduced. Therefore, by making cells for which the purity is less than a predetermined value (in this example, 75%) and the number of defects for which the result of visual classification and the result of automatic classification are the same is equal to or more than a predetermined value (in this example, 10) distinguishable, the sub class for which the purity should be improved can be made easily recognizable by the user. 
         [0097]    Next, the row of the sub class  5  is checked from right to left. Then, the sub class  5  includes numerical values in two rows of the main class A and the main class B, and it can be recognized that the defects are classified as the sub class  5  based on the sub classification model A and the sub classification model B. Further, it can be recognized that the sub class  5  has a purity of 100% for the sub classification model A of the main class A, which presents no problem, and has a purity as low as 57.9% for the sub classification model B of the main class B, and hence it can be recognized that the classification model that should be improved is the sub classification model B of the main class B. 
         [0098]    Therefore, by making cells included in the main class for which the purity is less than the predetermined value (in this example, 75%) and the number of defects for which the result of visual classification and the result of automatic classification are the same is equal to or more than the predetermined value (in this example, 10) distinguishable, the main class for which the purity should be improved can be made easily recognizable by the user. 
         [0099]    Next, the user focuses on the adjustment of the sub classification model B. The column of the main class B of this classification performance information table  145   a  includes values of the purity in the sub class  1 , the sub class  2 , and the sub class  5 , and hence it can be recognized that the sub classification model B is a model for classification into the three sub classes. When parameters of the sub classification model B are adjusted to improve the purity of the sub class  5 , the purities of the sub class  1  and the sub class  2  may be decreased. 
         [0100]    However, with respect to the sub class  1 , when the row of the sub class  1  is checked, it can be seen that the number of defects for which the result of visual classification and the result automatic classification are the same is as small as 4 in the main class B, and most defects in the sub class  1  are classified from the sub classification model C of the main class C. Similarly, the purity of the sub class  1  of the sub classification model C is as high as 100% and the total purity in the rightmost column of this table is also as high as 98.8%, and hence it can be anticipated that even when the parameters of the sub classification model B are adjusted, the effect on the purity of the sub class  1  is small. On the other hand, with respect to the sub class  2 , when the row of the sub class  2  is checked, it can be seen that the purity for the sub classification model A of the main class A is 93.9%, the purity for the sub classification model B of the main class B is 93.2%, the purity for the sub classification model C of the main class C is 75%, and the total purity of the sub class  2  is 94.2%. Further, the number of defects for which the result of visual classification and the result of automatic classification are the same is 31 for the sub classification model A, is 41 for the sub classification model B, and is 9 for the sub classification model C, and it can be recognized that the contribution of the sub classification model B is high. Therefore, it can be recognized that, when the parameters of the sub classification model B are adjusted, careful attention must be paid to a decrease in purity of the sub class  2 . 
         [0101]    Therefore, by making, of the cells included in the main class identified as described above, cells for which the number of defects for which the result of visual classification and the result of automatic classification are the same is equal to or more than the predetermined value (here, 10) distinguishable, the sub class for which the purity should be improved can be made easily recognizable by the user. 
         [0102]    As described above, through the processing by the extraction unit  150 , by just checking the classification performance information table  145   a , it can be recognized which sub classification model should be adjusted and, when the sub classification model is adjusted, what should be paid attention to. 
         [0103]    Note that, the classification performance information table  145   a  describes the purity, the accuracy, and the number of defects for which the result of visual classification and the result of automatic classification are the same in each cell, but the effect is not affected even when the accuracy is not described. Alternatively, a table of only the purity, and a table of only the number of defects for which the result of visual classification and the result of automatic classification are the same may be created separately. In addition, transposition of the rows and the columns is within the scope of the present invention. Alternatively, instead of displaying the table on the output unit  180 , the table may be processed internally to extract the sub classification models and the sub classes. 
         [0104]    Returning to  FIG. 2 , the adjustment unit  151  adjusts the sub classification model used by the classification unit  149  described above. Note that, the processing by the adjustment unit  151  is described in detail with reference to  FIG. 14 . 
         [0105]    The interface unit  153  transmits and receives information via the bus  190 . 
         [0106]    The classification recipe adjustment device  140  described above may be realized by a computer  900  as illustrated in, for example,  FIG. 13  (schematic diagram of the computer  900 ), including a central processing unit (CPU)  901 , a memory  902 , an external storage device  903  such as a hard disk drive (HDD), a read/write device  905  for reading and writing information from and to a portable recording medium  904  such as a compact disk (CD) or a digital versatile disk (DVD), an input device  906  such as a keyboard and a mouse, an output device  907  such as a display, and an I/F  908  for connecting to the bus. 
         [0107]    For example, the storage unit  141  may be realized when the CPU  901  uses the memory  902  or the external storage device  903 , the control unit  148  may be realized when a predetermined program stored in the external storage device  903  is loaded in the memory  902  and is executed by the CPU  901 , and the interface unit  153  may be realized when the CPU  901  uses the I/F  908 . 
         [0108]    The predetermined program may be downloaded from the recording medium  904  via the read/write device  905  or from a network via the I/F  908  to the external storage device  903 , and then loaded in the memory  902  to be executed by the CPU  901 . Alternatively, the predetermined program may be loaded directly from the recording medium  904  via the read/write device  905  or from the network via the I/F  908  to the memory  902  to be executed by the CPU  901 . 
         [0109]      FIG. 14  is an example of a flow chart illustrating the extraction processing by the extraction unit  150  and the processing of adjusting the sub classification model by the adjustment unit  151 . 
         [0110]    First, the extraction unit  150  extracts, from the classification performance information table  145   a  stored in the classification performance information storage region  145 , the sub class to be adjusted, and extracts the sub classification model corresponding to the sub class (S 10 ). 
         [0111]    For example, the extraction unit  150  identifies, in the aggregate column  145   g  of the classification performance information table  145   a , and in a row including cells for which the value of the purity is less than 75% and the number of defects for which the result of visual classification and the result of automatic classification are the same is 10 or more, from among the main classes in which the value of the purity is stored, a main class for which the value of the purity is less than 75% and the number of defects for which the result of visual classification and the result of automatic classification are the same is 10 or more. Further, the extraction unit  150  identifies, from among cells included in the identified main class, a cell for which the number of defects for which the result of visual classification and the result of automatic classification are the same is 10 or more, and identifies the sub class corresponding to the identified cell from the sub class column  145   b . The main class and the sub class, which are identified as described above, are extracted as adjustment targets. In the classification performance information table  145   a  illustrated in  FIG. 6 , through the above-mentioned processing, the sub class  2  and the sub class  5  are extracted as the adjustment targets. 
         [0112]    Next, the adjustment unit  151  extracts weight coefficients corresponding to the main class and the sub class extracted in Step S 11  as the adjustment targets from the weight coefficient information storage region  143  (S 11 ). 
         [0113]    For example, the adjustment unit  151  extracts, from the weight coefficient information table  143   a  as illustrated in  FIG. 4 , the weight coefficients stored in the row of the sub class  2  and the row of the sub class  5 . 
         [0114]    Here, in this embodiment, from the row of the sub class  2 , the weight coefficient having the largest value and the weight coefficient having the second largest value are extracted. Specifically, the weight coefficient of 0.29 at k=5 and the weight coefficient of 0.26 at k=6 are extracted. Similarly, from the row of the sub class  5 , the weight coefficient having the largest value and the weight coefficient having the second largest value are extracted. Specifically, the weight coefficient of 0.33 at k=7 and the weight coefficient of 0.23 at k=3 are extracted. Note that, the number of the extracted weight coefficients may be a predetermined number, or all the weight coefficients stored in the target row may be extracted. 
         [0115]    Next, the adjustment unit  151  changes values of the weight coefficients extracted in Step S 11  and reevaluates the classification performance (S 12 ). 
         [0116]    For example, in this embodiment, the adjustment unit  151  generates an orthogonal table in design of experiments. With the extracted four types of weight coefficients as factors, the orthogonal table is generated with three or five levels. In this embodiment, two types of weight coefficients are extracted from each of two sub classes, and hence four types of weight coefficients are used as the factors. However, six types of weight coefficients may be used when there are three sub classes to be adjusted, and eight types of weight coefficients may be used when there are four sub classes to be adjusted. Further, which of three levels, five levels, or more levels are used may be determined in advance. 
         [0117]      FIG. 15  is a schematic diagram of an orthogonal table  146   a  generated with four types of extracted weight coefficients as the factors and with three levels. 
         [0118]    As illustrated in the figure, the orthogonal table  146   a  includes a trial ID column  146   b , a W B,2,5  column  146   c , a W B,2,6  column  146   d , a W B,5,3  column  146   e , and a W B,5,7  column  146   f.    
         [0119]    The trial ID column  146   b  stores identification information capable of identifying an experiment identified in each row. In this case, consecutive natural numbers are allocated sequentially to the rows from above as trial IDs. 
         [0120]    The W B,2,5  column  146   c  stores information for identifying a value to be added to or subtracted from a weight coefficient W B,2,5  used to the attribute data X 5  extracted in Step S 11 , of the weight coefficients of the main class B and the sub class  2  extracted in Step S 10 . 
         [0121]    The W 3,2,6  column  146   d  stores information for identifying a value to be added to or subtracted from a weight coefficient W B,2,6  used to the attribute data X 6  extracted in Step S 11 , of the weight coefficients of the main class B and the sub class  2  extracted in Step S 10 . 
         [0122]    The W B,5,3  column  146   e  stores information for identifying a value to be added to or subtracted from a weight coefficient W B,5,3  used to the attribute data X 3  extracted in Step S 11 , of the weight coefficients of the main class B and the sub class  5  extracted in Step S 10 . 
         [0123]    The W B,5,7  column  146   f  stores information for identifying a value to be added to or subtracted from a weight coefficient W B,5,7  used to the attribute data X 7  extracted in Step S 11 , of the weight coefficients of the main class B and the sub class  5  extracted in Step S 10 . 
         [0124]    Here, in this embodiment, the original weight coefficient (±0), a weight coefficient obtained by adding 0.1 to the original weight coefficient, and a weight coefficient obtained by subtracting 0.1 from the original weight coefficient are prepared to perform each experiment (reexamination of the classification performance). That is, in the above description, the change amount of the weight coefficient is 0.1, but the change amount is not limited to 0.1 and another value may be set in advance. 
         [0125]    Note that, the thus generated orthogonal table  146   a  is stored in the adjustment information storage region  146 . 
         [0126]    Then, the adjustment unit  151  changes the value of the weight coefficient for each trial ID stored in the trial ID column  146   b  of the orthogonal table  146   a  to classify the sub classes again by using Equation (1) described above, and compares the result of classification with the result of visual classification received as an input via the input unit  170 , to thereby calculate values of the purity and the accuracy of each of the sub class  1 , the sub class  2 , and the sub class  3  in the main class B and calculate values of the purity and the accuracy of each of the sub class  1 , the sub class  2 , and the sub class  3  in all the main classes. 
         [0127]    The thus calculated values of the purity and the accuracy in the main class and total values of the purity and the accuracy are stored in a reevaluation table  146   g  in the main class as illustrated in  FIG. 16  (schematic diagram of the reevaluation table  146   g  in the main class) and a reevaluation table  146   q  in total as illustrated in  FIG. 17  (schematic diagram of the reevaluation table  146   q  in total). 
         [0128]    As illustrated in  FIG. 16 , the reevaluation table  146   g  in the main class includes a trial ID column  146   h , a main class purity column  146   i , and a main class accuracy column  146   m , the main class purity column  146   i  includes a sub class  1  column  146   j , a sub class  2  column  146   k , and a sub class  5  column  146   l , and the main class accuracy column  146   m  includes a sub class  1  column  146   n , a sub class  2  column  146   o , and a sub class  5  column  146   p.    
         [0129]    The trial ID column  146   h  stores a trial ID corresponding to the trial ID stored in the trial ID column  146   b  of the orthogonal table  146   a.    
         [0130]    The sub class  1  column  146   j  of the main class purity column  146   i  stores information for identifying the value of the purity of the main class (in this example, the main class B) to be inspected for the number of defects classified as the sub class  1  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   h . Note that, the method of calculating the value of the purity is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  1  of the main class B column  145   d  of the classification performance information table  145   a.    
         [0131]    The sub class  2  column  146   k  of the main class purity column  146   i  stores information for identifying the value of the purity of the main class (in this example, the main class B) to be inspected for the number of defects classified as the sub class  2  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   h . Note that, the method of calculating the value of the purity is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  2  of the main class B column  145   d  of the classification performance information table  145   a.    
         [0132]    The sub class  5  column  146   l  of the main class purity column  146   i  stores information for identifying the value of the purity of the main class (in this example, the main class B) to be inspected for the number of defects classified as the sub class  5  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   h . Note that, the method of calculating the value of the purity is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  5  of the main class B column  145   d  of the classification performance information table  145   a.    
         [0133]    The sub class  1  column  146   n  of the main class accuracy column  146   m  stores information for identifying the value of the accuracy of the main class (in this example, the main class B) to be inspected for the number of defects classified as the sub class  1  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   h . Note that, the method of calculating the value of the accuracy is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  1  of the main class B column  145   d  of the classification performance information table  145   a.    
         [0134]    The sub class  2  column  146   o  of the main class accuracy column  146   m  stores information for identifying the value of the accuracy of the main class (in this example, the main class B) to be inspected for the number of defects classified as the sub class  2  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   h . Note that, the method of calculating the value of the accuracy is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  2  of the main class B column  145   d  of the classification performance information table  145   a.    
         [0135]    The sub class  5  column  146   p  of the main class accuracy column  146   m  stores information for identifying the value of the accuracy of the main class (in this example, the main class B) to be inspected for the number of defects classified as the sub class  5  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   h . Note that, the method of calculating the value of the accuracy is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  5  of the main class B column  145   d  of the classification performance information table  145   a.    
         [0136]    As illustrated in  FIG. 17 , the reevaluation table in total  146   q  includes a trial ID column  146   r , a total purity column  146   s , and a total accuracy column  146   w , the total purity column  146   s  includes a sub class  1  column  146   t , a sub class  2  column  146   u , and a sub class  5  column  146   v , and the total accuracy column  146   w  includes a sub class  1  column  146   x , a sub class  2  column  146   y , and a sub class  5  column  146   z.    
         [0137]    The trial ID column  146   r  stores a trial ID corresponding to the trial ID stored in the trial ID column  146   b  of the orthogonal table  146   a.    
         [0138]    The sub class  1  column  146   t  of the total purity column  146   s  stores information for identifying the value of the purity in all the main classes of the number of defects classified as the sub class  1  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   r . Note that, the method of calculating the value of the purity is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  1  of the aggregate column  145   g  of the classification performance information table  145   a.    
         [0139]    The sub class  2  column  146   u  of the total purity column  146   s  stores information for identifying the value of the purity in all the main classes of the number of defects classified as the sub class  2  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   r . Note that, the method of calculating the value of the purity is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  2  of the aggregate column  145   g  of the classification performance information table  145   a.    
         [0140]    The sub class  5  column  146   v  of the total purity column  146   s  stores information for identifying the value of the purity in all the main classes of the number of defects classified as the sub class  5  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   r . Note that, the method of calculating the value of the purity is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  5  of the aggregate column  145   g  of the classification performance information table  145   a.    
         [0141]    The sub class  1  column  146   x  of the total accuracy column  146   w  stores information for identifying the value of the accuracy in all the main classes of the number of defects classified as the sub class  1  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   r . Note that, the method of calculating the value of the accuracy is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  1  of the aggregate column  145   g  of the classification performance information table  145   a.    
         [0142]    The sub class  2  column  146   y  of the total accuracy column  146   w  stores information for identifying the value of the accuracy in all the main classes of the number of defects classified as the sub class  2  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   r . Note that, the method of calculating the value of the accuracy is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  2  of the aggregate column  145   g  of the classification performance information table  145   a.    
         [0143]    The sub class  5  column  146   z  of the total accuracy column  146   w  stores information for identifying the value of the accuracy in all the main classes of the number of defects classified as the sub class  5  by the weight coefficient in the row of the orthogonal table  146   a  in which the trial ID column  146   b  stores the trial ID identified in the trial ID column  146   r . Note that, the method of calculating the value of the accuracy is the same as the method used in calculating the value to be stored in the cell corresponding to the sub class  5  of the aggregate column  145   g  of the classification performance information table  145   a.    
         [0144]    Returning to  FIG. 14 , the adjustment unit  151  creates a factor effect diagram from the orthogonal table  146   a , the reevaluation table  146   g  in the main class, and the reevaluation table  146   q  in total, which are stored in the adjustment information storage region  146 , to be output to the output unit  180  (S 13 ). 
         [0145]    For example, the adjustment unit  151  creates at least one of a first factor effect diagram  196   a  as illustrated in  FIG. 18  (schematic diagram of the first factor effect diagram  196   a ) and a second factor effect diagram  197   a  as illustrated in  FIG. 19  (schematic diagram of the second factor effect diagram  197   a ) to be output to the output unit  180 . 
         [0146]    The first factor effect diagram  196   a  illustrated in  FIG. 18  is a factor effect diagram of the purities of the main class B, that is, the sub classification model B, and includes a sub class  1  display region  196   b , a sub class  2  display region  196   c , and a sub class  5  display region  196   d.    
         [0147]    In each of the display regions  196   b ,  196   c , and  196   d , for each of levels (“−0.1”, “±0”, and “+0.1”) of the weight coefficients, an average value of the purities of the level is plotted, and the average values of the three levels are expressed as a polygonal line by being connected by lines for each weight coefficients. 
         [0148]    For example, a polygonal line  196   e  expresses the purities of the sub class  1  in the sub classification model B with respect to the three levels of the weight coefficient to the sixth attribute data X 6  of the sub class  2  in the sub classification model B. When the level of ±0 has the largest value of the three plotted points as in the polygonal line  196   e , it means that the original weight coefficient is the optimal solution. 
         [0149]    Further, the adjustment unit  151  calculates significance probability, which is obtained by executing analysis of variance on each weight coefficient, to be displayed in a significance probability display region  196   f . In this case, as the value of the significance probability becomes smaller, there is a larger statistical difference between the plotted points of the polygonal line of the target weight coefficient, and hence it can be recognized that the effect on the purities becomes larger. 
         [0150]    The second factor effect diagram  197   a  illustrated in  FIG. 19  is a factor effect diagram of the purities for each weight coefficient, and includes a weight coefficient W B,2,5  display region  197   b , a weight coefficient W B,2,6  display region  197   c , a weight coefficient W B,5,3  display region  197   d , and a weight coefficient W B,5,7  display region  197   e.    
         [0151]    In each of the display regions  197   b ,  197   c ,  197   d , and  197   e , a polygonal line obtained by plotting and connecting by lines average values of the purities of three levels for each sub class in the main class (in this case, the main class B) as the adjustment target, and a polygonal line obtained by plotting and connecting by lines the average values of the purities of the three levels for each sub class in all the main classes (total). 
         [0152]    Therefore, with the first factor effect diagram  196   a  illustrated in  FIG. 18 , it cannot be recognized how the overall purities change. However, with the second factor effect diagram  197   a  illustrated in  FIG. 19 , the sub classification model B and the overall purities can be compared, and when the weight coefficient to the seventh attribute data of the sub class  5  of the sub classification model B is changed to the weight coefficient obtained by adding 0.1 thereto, it can be easily recognized that the overall purity of the sub class  5  can be improved. 
         [0153]    Returning to  FIG. 14 , the adjustment unit  151  determines the value of the weight coefficient to be adjusted from the factor effect diagrams of  FIG. 18  and  FIG. 19  (S 14 ). 
         [0154]    For example, the adjustment unit  151  adjusts, in the second factor effect diagram  197   a  of  FIG. 19 , the weight coefficient of the adjustment target to be the weight coefficient that exhibits highest average value of the purities of the three levels for each sub class in all the main classes (total). 
         [0155]    As describe above, the defect classification apparatus  100  in this embodiment has functions of extracting the classification model and the sub class to be effectively adjusted, changing the weight coefficient corresponding to the extracted classification model and sub class, and adjusting the classification performance, with the result that the number of classes for which the result of automatic classification can be trusted may be increased and the number of steps of the visual classification by a person may be decreased. 
         [0156]    Note that, this embodiment has been described with reference to the defect classification apparatus  100  including the SEM  110 . However, the present invention is not limited to the SEM  110 , and any detector may be used as long as an image of various wavelengths such as visible light, ultraviolet ray, and infrared ray may be taken. Further, the SEM  110  may be in a form separated from the defect classification apparatus  100  (separate casing or the like). 
         [0157]    Further, this embodiment has been described with reference to the classification model having the two stages of the main classification model and the sub classification model. However, the present invention is also effectively applicable to first and second stages or second and third stages of a classification model having three stages. 
       REFERENCE SIGNS LIST 
       [0000]    
       
         
           
               100  defect classification apparatus 
               110  SEM 
               130  general control unit 
               140  classification recipe adjustment device 
               141  storage unit 
               142  defect information storage region 
               143  weight coefficient information storage region 
               144  likelihood information storage region 
               145  classification performance information storage region 
               146  adjustment information storage region 
               148  control unit 
               149  classification unit 
               150  extraction unit 
               151  adjustment unit 
               153  interface unit 
               160  secondary storage device 
               170  input unit 
               180  output unit