Abstract:
The present invention relates to a clustering method based on iterations of neural networks, which comprises the following steps: step 1, initializing parameters of an extreme learning machine; step 2, randomly choosing samples of which number is equal to the number of clusters, each sample representing one cluster, forming an initial exemplar set and training the extreme learning machine; step 3, using current extreme learning machine to cluster samples, which generates a clustering result; step 4, choosing multiple samples from each cluster as exemplars for the cluster according to a rule; step 5, retraining the extreme learning machine by using the exemplars for each cluster obtained from step 4; and step 6, going back to step 3 to do iteration, otherwise obtaining and outputting clustering result until clustering result is steady or a maximal limit of the number of iterations is reached. The present invention resolves problems that how to realize clustering of high dimensional and nonlinear data space and that the prior art consumes a larger memory or need longer running time.

Description:
TECHNICAL FIELD 
       [0001]    The present invention generally relates to a method for resolving classic clustering issues in machine learning, in particular to an extreme learning machine algorithm which is an improvement of neural networks. 
       BACKGROUND 
       [0002]    Data clustering is always an important research in engineering systems and computational science. In this age of big data with information explosion, business requirement related to data processing is increased day by day, and data to be processed is becoming more and more complicated. As intensive study of clustering is made, new clustering methods are continuously proposed. Traditional clustering algorithm, such as k-means clustering algorithm, expectation maximization algorithm and hierarchical clustering algorithm, is fast, but only adapts to situation of which data space is linearly separable. New clustering algorithm, such as spectral clustering algorithm, deep autoencoder, and unsupervised extreme learning machine, can resolve clustering issues in high dimensional and linearly inseparable data space, but generally need to consume a larger memory or need longer running time. 
         [0003]    At present, to cluster large-scale and complex data, clustering algorithm is required to be capable of processing high dimensional and linearly inseparable data and reduce cost of memory and running time. The present invention resolves clustering issues by neural networks to effectively process high dimensional and nonlinear complex data and solves neural networks by using extreme learning machine to greatly improve running speed of the clustering algorithm. The extreme learning machine is a learning method of single hidden layer feed-forward network and is proposed by Professor Guangbin Huang in 2004. Different from improved gradient descent method of traditional neural networks, learning process of the extreme learning machine doesn&#39;t need iterations, which greatly reduces training time. Besides, the present invention takes k-means algorithm as a basic framework, so that parallelizing means for accelerating can be used to reduce time cost of clustering. And the present invention is based on neural networks with the extreme learning machine and only uses a small amount of exemplar sets to train, therefore shorter computing time and less memory consumption is needed. 
         [0004]    In general, the present invention resolves issues that traditional clustering algorithm can&#39;t properly process clustering problem of high dimensional and nonlinear data space and that new clustering algorithm consumes a larger memory or need longer running time and is capable of effectively processing clustering problem of large-scale and complex data. 
       SUMMARY 
       [0005]    The present invention is to provide a clustering method based on iterations of neural networks. The present invention is based on a framework of traditional clustering algorithm, uses neural networks on similarity measurement and optimizes network structure continuously to get a more accurate similarity measurement. 
         [0006]    The present invention may be implemented by: a clustering method based on iterations of neural networks comprising the following steps: 
         [0007]    step 1, initializing parameters of an extreme learning machine; 
         [0008]    step 2, randomly choosing samples of which number is equal to the number of clusters, each sample representing one cluster; and training the extreme learning machine to obtain an initial output weights between hidden-layer and output-layer; 
         [0009]    step 3, using current extreme learning machine to cluster samples; 
         [0010]    step 4, choosing multiple samples from each cluster as exemplars for the cluster according to a rule; 
         [0011]    step 5, retraining the extreme learning machine by using the exemplars for each cluster obtained from previous step; and 
         [0012]    step 6, outputting clustering result if termination condition is met, otherwise going back to step 3. 
         [0013]    Preferably, step 1 comprises two substeps: 
         [0014]    substep 1.1, setting a number of hidden-layer neurons L, an activation function of hidden-layer g(θ), and a regularization factor γ of the extreme learning machine, wherein the number of hidden-layer neurons L is greater than a feature dimension d of sample, the activation function of hidden-layer g(θ) usually adopts Sigmoid function, and value range of the regularization factor γ is generally [10 −5 , 10 10 ]; and 
         [0015]    substep 1.2, randomly initializing weights between input-layer and hidden-layer W and a bias b of the extreme learning machine, wherein the weights between input-layer and hidden-layer W is a random matrix with d rows and L columns whose random range is [−1,1], and the bias b is a random matrix with 1 row and L columns whose random range is [−1,1], one of columns of the bias corresponding to a bias of a hidden-layer neuron. 
         [0016]    Preferably, step 2 comprises two substeps: 
         [0017]    substep 2.1, randomly choosing samples of which number is equal to a number of clusters, that is, choosing K samples, each sample representing one cluster, so as to form an initial exemplar set and corresponding target output to train the extreme learning machine; and 
         [0018]    substep 2.2, calculating an output of the hidden-layer neurons H (0)  and output weights between hidden-layer and output-layer β (0)  to obtain an initial model of the extreme learning machine through Eq. (2), Eq. (3) and Eq. (4): 
         [0000]    
       
         
           
             
               
                 
                   
                     H 
                     
                       ( 
                       0 
                       ) 
                     
                   
                   = 
                   
                     g 
                      
                     
                       ( 
                       
                         
                           W 
                           · 
                           
                             E 
                             
                               ( 
                               0 
                               ) 
                             
                           
                         
                         + 
                         
                           
                             B 
                              
                             
                                 
                             
                           
                           
                             ( 
                             0 
                             ) 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       if 
                        
                       
                           
                       
                        
                       K 
                     
                     &gt; 
                     L 
                   
                   , 
                   
                     
                       β 
                       
                         ( 
                         0 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ( 
                           
                             
                               
                                 H 
                                 
                                   
                                     ( 
                                     0 
                                     ) 
                                   
                                   T 
                                 
                               
                                
                               
                                 H 
                                 
                                   ( 
                                   0 
                                   ) 
                                 
                               
                             
                             + 
                             
                               
                                 I 
                                 L 
                               
                               γ 
                             
                           
                           ) 
                         
                         † 
                       
                        
                       
                         H 
                         
                           
                             ( 
                             0 
                             ) 
                           
                           T 
                         
                       
                        
                       
                         T 
                         
                           ( 
                           0 
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
             
               
                 
                   
                     else 
                      
                     
                         
                     
                      
                     
                       β 
                       
                         ( 
                         0 
                         ) 
                       
                     
                   
                   = 
                   
                     
                       
                         
                           H 
                           
                             
                               ( 
                               0 
                               ) 
                             
                             T 
                           
                         
                          
                         
                           ( 
                           
                             
                               
                                 H 
                                 
                                   ( 
                                   0 
                                   ) 
                                 
                               
                                
                               
                                 H 
                                 
                                   
                                     ( 
                                     0 
                                     ) 
                                   
                                   T 
                                 
                               
                             
                             + 
                             
                               
                                 I 
                                 K 
                               
                               γ 
                             
                           
                           ) 
                         
                       
                       † 
                     
                      
                     
                       T 
                       
                         ( 
                         0 
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
         [0000]    wherein B (0)  is a row-expansive matrix of the bias b of which number of rows is equal to that of E (0) , which is a matrix with K rows and L columns, K being the number of clusters, and L being the number of hidden-layer neurons, H (0)     T    represents a transpose of the matrix H (0) , (•) †  represents a generalized inverse matrix, I L  is an identity matrix of L dimension, and I K  is an identity matrix of K dimension. 
         [0019]    Preferably, step 3 comprises two substeps: 
         [0020]    Substep 3.1, calculating an output value of the sample x i  on an output-layer neuron o i  by using parameters of the extreme learning machine currently used through Eq. (5) and Eq. (6), wherein the weights between input-layer and hidden-layer W, the bias b, the activation function of hidden-layer g(θ), the number of hidden-layer neurons L and the regularization factor γ are changeless during iterations, while the output weights between hidden-layer and output-layer β continues to change as iterations progress, Eq. (5) and Eq. (6) being as follows: 
         [0000]        h   i   =g ( Wx   i   +b )  (5)
 
         [0000]        o   i   =h   i β (m-1)   (6)
 
         [0000]    wherein h i  represents an output value of the sample x i  on the hidden-layer and is a matrix with 1 row and L columns, of represents the output value of the sample x i  on the output-layer and is a matrix with 1 row and K columns, and β (m-1)  represents output weights between hidden-layer and output-layer obtained in the previous iteration; and 
         [0021]    substep 3.2, clustering the samples according to the output value of the samples on the output-layer neurons, wherein assigning rule is that a sample should be assigned to a cluster corresponding to the neuron with the highest output value of the output-layer neurons on basis of which the number of the output-layer neurons is equal to the number of clusters. 
         [0022]    Preferably, step 4 comprises two substeps: 
         [0023]    substep 4.1, calculating each cluster mean mean according to clustering result of step 3; and 
         [0024]    substep 4.2, calculating distances between each sample of the cluster and a cluster center in each cluster, and choosing ρ samples which are the nearest to the cluster center to form an exemplar set of current iteration E (m) , wherein if the number of samples of a cluster is less than the number of exemplars ρ required to be selected from the cluster, all samples of the cluster are selected to be exemplars for the cluster; and if a cluster doesn&#39;t contain any sample, a sample as a exemplar for the cluster should be randomly selected from all samples with exception of exemplar sets for all the other clusters. 
         [0025]    Preferably, step 5 comprises two substeps: 
         [0026]    substep 5.1, constructing target functions for exemplar sets obtained from step 4; and 
         [0027]    substep 5.2, calculating an output of the hidden-layer neurons H (m)  and output weights between hidden-layer and output-layer β (m)  of the extreme learning machine after being retrained through Eq. (7), Eq. (8) and Eq. (9): 
         [0000]    
       
         
           
             
               
                 
                   
                     H 
                     
                       ( 
                       m 
                       ) 
                     
                   
                   = 
                   
                     g 
                      
                     
                       ( 
                       
                         
                           W 
                           · 
                           
                             E 
                             
                               ( 
                               m 
                               ) 
                             
                           
                         
                         + 
                         
                           B 
                           
                             ( 
                             m 
                             ) 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       if 
                        
                       
                           
                       
                        
                       K 
                       × 
                       ρ 
                     
                     &gt; 
                     L 
                   
                   , 
                   
                     
                       β 
                       
                         ( 
                         m 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ( 
                           
                             
                               
                                 H 
                                 
                                   
                                     ( 
                                     m 
                                     ) 
                                   
                                   T 
                                 
                               
                                
                               
                                 H 
                                 
                                   ( 
                                   m 
                                   ) 
                                 
                               
                             
                             + 
                             
                               
                                 I 
                                 L 
                               
                               γ 
                             
                           
                           ) 
                         
                         † 
                       
                        
                       
                         H 
                         
                           
                             ( 
                             m 
                             ) 
                           
                           T 
                         
                       
                        
                       
                         T 
                         
                           ( 
                           m 
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
             
               
                 
                   
                     else 
                      
                     
                         
                     
                      
                     
                       β 
                       
                         ( 
                         m 
                         ) 
                       
                     
                   
                   = 
                   
                     
                       
                         
                           H 
                           
                             
                               ( 
                               m 
                               ) 
                             
                             T 
                           
                         
                          
                         
                           ( 
                           
                             
                               
                                 H 
                                 
                                   ( 
                                   m 
                                   ) 
                                 
                               
                                
                               
                                 H 
                                 
                                   
                                     ( 
                                     m 
                                     ) 
                                   
                                   T 
                                 
                               
                             
                             + 
                             
                               
                                 I 
                                 
                                   K 
                                   × 
                                   ρ 
                                 
                               
                               γ 
                             
                           
                           ) 
                         
                       
                       † 
                     
                      
                     
                       T 
                       
                         ( 
                         m 
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
         [0000]    wherein B (m)  is a row-expansive matrix of the bias b of which number of rows is equal to that of E (m) , which is a matrix with K×ρ rows and L columns, K being the number of clusters, ρ being the number of exemplars of each cluster, and L being the number of hidden-layer neurons. H (m)     T    represents a transpose of the matrix H (m) , (•) †  represents the generalized inverse matrix, I L  is the identity matrix of L dimension, and I K×ρ  is an identity matrix of K×ρ dimension. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0028]      FIG. 1  is a frame diagram of an extreme learning machine; 
           [0029]      FIG. 2  is a flow diagram of a clustering algorithm based on iterations of neural networks of the present invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0030]    Below is a detailed description to further describe the present invention. 
         [0031]    The present invention specifies a sample set to be clustered D={x i |x i εR d ,i=1, . . . , N} and a number of clusters K, wherein d is a feature dimension of sample, N is a number of samples, and x i  is a d-dimension eigenvectors which is a matrix with 1 row and d columns. We here takes a sample set to be clustered implemented by Iris dataset of UCI as an example, wherein the number of clusters K is 3, the feature dimension d of sample is 4, the number of samples N is 150, and x i  is a matrix with 1 row and 4 columns. 
         [0032]    Firstly, the present invention completes task that initializes required parameter of an extreme learning machine in step 1, wherein step 1 comprises two substeps: 
         [0033]    Substep 1.1, the present invention sets a number of hidden-layer neurons L, an activation function of hidden-layer g(θ), and a regularization factor γ of the extreme learning machine, wherein the number of hidden-layer neurons L is generally greater than the feature dimension d of sample. In the present invention, the number of hidden-layer neurons has little impact on final clustering result, and is generally 1000 or 2000. The activation function of hidden-layer g(θ) usually adopts Sigmoid function, wherein the Sigmoid function is indicated in Eq.(1): 
         [0000]    
       
         
           
             
               
                 
                   
                     g 
                      
                     
                       ( 
                       ϑ 
                       ) 
                     
                   
                   = 
                   
                     1 
                     
                       1 
                       + 
                       
                         g 
                         
                           - 
                           ϑ 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0034]    If θ is a matrix, Eq. (1) is interpreted to calculate Sigmoid function for each element of the matrix. Value range of the regularization factor γ is generally [10 −5 , 10 10 ], and value of the regularization factor can be slightly adjusted according to different datasets. In this embodiment, the number of hidden-layer neurons L is 1000, the activation function of hidden-layer g(θ) is the Sigmoid function, and the regularization factor γ is 108. 
         [0035]    Substep 1.2, the present invention randomly initializes weights between input-layer and hidden-layer W and a bias b of the extreme learning machine. 
         [0036]    Since both the number of the input-layer neurons and the feature dimension of sample are d, and the number of hidden-layer neurons is L, the weights between input-layer and hidden-layer W is a random matrix with d rows and L columns whose random range is generally [−1,1], and the bias b is a random matrix with 1 row and L columns whose random range is generally [−1,1], one of columns of the bias corresponding to a bias of a hidden-layer neuron. In this embodiment, the weights between input-layer and hidden-layer W is a random matrix with 3 rows and 1000 columns whose random range is [−1,1], and the bias b is a random matrix with 1 row and 1000 columns whose random range is [−1,1]. 
         [0037]    It should be noted that data is required to be normalized while using learning model of neural networks. Linear function, or zero-mean standardization which could map data into [0,1] is generally adopted to normalize. This embodiment adopts linear function to normalize. 
         [0038]    Next, an initial model of the extreme learning machine is obtained by step 2 which comprises two substeps: 
         [0039]    Substep 2.1, the present invention randomly selects samples of which number is equal to the number of clusters, that is, selects K samples, each sample representing one cluster, so that an initial exemplar set E (0)  and corresponding target output T (0)  are formed to train the extreme learning machine. It should be noted that training a neural network requires to generate a target output for each sample for which what is usually to do is to set the bit that represents classification of a sample into “1” and other bits into “0”. Taking 3 randomly selected samples from Iris dataset of UCI as an example, as presented in Table 1: 
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 table for the samples and corresponding target outputs 
               
             
          
           
               
                   
                 Target 
                 Classification 
               
               
                 Initial exemplar set E (0)   
                 output T (0)   
                 of a sample 
               
               
                   
               
               
                 [0.54, 0.47, 0.54, 0.60](random sample 1) 
                 [1 0 0] 
                 First class 
               
               
                 [0.66, 0.54, 0.72, 0.91] (random sample 2) 
                 [0 1 0] 
                 Second class 
               
               
                 [0.13, 0.58, 0.10, 0.05] (random sample 3) 
                 [0 0 1] 
                 Third class 
               
               
                   
               
             
          
         
       
     
         [0040]    In above example, the initial exemplar sets E (0)  can also be presented as 
         [0000]    
       
         
           
             
               
                 E 
                 
                   ( 
                   0 
                   ) 
                 
               
               = 
               
                 [ 
                 
                   
                     
                       0.54 
                     
                     
                       0.47 
                     
                     
                       0.54 
                     
                     
                       0.60 
                     
                   
                   
                     
                       0.66 
                     
                     
                       0.54 
                     
                     
                       0.72 
                     
                     
                       0.91 
                     
                   
                   
                     
                       0.13 
                     
                     
                       0.58 
                     
                     
                       0.10 
                     
                     
                       0.05 
                     
                   
                 
                 ] 
               
             
             , 
           
         
       
     
         [0000]    and the target outputs T (0)  as 
         [0000]    
       
         
           
             
               T 
               
                 ( 
                 0 
                 ) 
               
             
             = 
             
               
                 [ 
                 
                   
                     
                       1 
                     
                     
                       0 
                     
                     
                       0 
                     
                   
                   
                     
                       0 
                     
                     
                       1 
                     
                     
                       0 
                     
                   
                   
                     
                       0 
                     
                     
                       0 
                     
                     
                       1 
                     
                   
                 
                 ] 
               
               . 
             
           
         
       
     
         [0041]    Substep 2.2, the present invention calculates an output of the hidden-layer neurons H (0)  and weights of the hidden-layer relative to an output-layer β (0)  to obtain the initial model of the extreme learning machine through Eq. (2), Eq. (3) and Eq. (4): 
         [0000]    
       
         
           
             
               
                 
                   
                     H 
                     
                       ( 
                       0 
                       ) 
                     
                   
                   = 
                   
                     g 
                      
                     
                       ( 
                       
                         
                           W 
                           · 
                           
                             E 
                             
                               ( 
                               0 
                               ) 
                             
                           
                         
                         + 
                         
                           
                             B 
                              
                             
                                 
                             
                           
                           
                             ( 
                             0 
                             ) 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       if 
                        
                       
                           
                       
                        
                       K 
                     
                     &gt; 
                     L 
                   
                   , 
                   
                     
                       β 
                       
                         ( 
                         0 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ( 
                           
                             
                               
                                 H 
                                 
                                   
                                     ( 
                                     0 
                                     ) 
                                   
                                   T 
                                 
                               
                                
                               
                                 H 
                                 
                                   ( 
                                   0 
                                   ) 
                                 
                               
                             
                             + 
                             
                               
                                 I 
                                 L 
                               
                               γ 
                             
                           
                           ) 
                         
                         † 
                       
                        
                       
                         H 
                         
                           
                             ( 
                             0 
                             ) 
                           
                           T 
                         
                       
                        
                       
                         T 
                         
                           ( 
                           0 
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
             
               
                 
                   
                     else 
                      
                     
                         
                     
                      
                     
                       β 
                       
                         ( 
                         0 
                         ) 
                       
                     
                   
                   = 
                   
                     
                       
                         
                           H 
                           
                             
                               ( 
                               0 
                               ) 
                             
                             T 
                           
                         
                          
                         
                           ( 
                           
                             
                               
                                 H 
                                 
                                   ( 
                                   0 
                                   ) 
                                 
                               
                                
                               
                                 H 
                                 
                                   
                                     ( 
                                     0 
                                     ) 
                                   
                                   T 
                                 
                               
                             
                             + 
                             
                               
                                 I 
                                 K 
                               
                               γ 
                             
                           
                           ) 
                         
                       
                       † 
                     
                      
                     
                       T 
                       
                         ( 
                         0 
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
         [0000]    wherein B (0)  is a row-expansive matrix of the bias b of which number of rows is equal to that of E (0) , which is a matrix with K rows and L columns, K being the number of clusters, L being the number of hidden-layer neurons, H (0)     T    represents a transpose of the matrix H (0) , (•) †  represents a generalized inverse matrix, I L  is an identity matrix of L dimension, and I K  is an identity matrix of K dimension. 
         [0042]    For instance, when the number of hidden-layer neurons is 1000 and 3 samples are randomly selected, the bias b and its row-expansive matrix B (0)  are: 
         [0000]    
       
         
           
             
               b 
               = 
               
                 
                   [ 
                   
                     
                       
                         0.033 
                       
                       
                         … 
                       
                       
                         
                           - 
                           0.334 
                         
                       
                       
                         0.125 
                       
                     
                   
                   ] 
                 
                 
                   1 
                   × 
                   1000 
                 
               
             
             , 
             
               
 
             
              
             
               
                 B 
                 
                   ( 
                   0 
                   ) 
                 
               
               = 
               
                 
                   [ 
                   
                     
                       
                         0.033 
                       
                       
                         … 
                       
                       
                         
                           - 
                           0.334 
                         
                       
                       
                         0.125 
                       
                     
                     
                       
                         0.033 
                       
                       
                         … 
                       
                       
                         
                           - 
                           0.334 
                         
                       
                       
                         0.125 
                       
                     
                     
                       
                         0.033 
                       
                       
                         … 
                       
                       
                         0.334 
                       
                       
                         0.125 
                       
                     
                   
                   ] 
                 
                 
                   3 
                   × 
                   1000 
                 
               
             
           
         
       
     
         [0043]    Below is iteration part, wherein m represents a number of iterations with an initial value of 1. The present invention uses a extreme learning machine of previous iteration to cluster the samples in step 3 which comprises two substeps: 
         [0044]    Substep 3.1, in the extreme learning machine used in following calculation, the weights between input-layer and hidden-layer W, the bias b, the activation function of hidden-layer g(θ), the number of hidden-layer neurons L and the regularization factor γ are changeless during iterations, while the output weights between hidden-layer and output-layer β continues to change as iterations progress. Output value of a sample on an output-layer neuron is calculated by using parameters of current extreme learning machine through Eq. (5) and Eq. (6): 
         [0000]        h   i   =g ( Wx   i   +b )  (5)
 
         [0000]        o   i   =h   i β (m-1)   (6)
 
         [0000]    wherein h i  represents an output value of the sample x i  on the hidden-layer and is a matrix with 1 row and L columns, o i  represents the output value of the sample x i  on the output-layer and is a matrix with 1 row and K columns, and β (m-1)  represents output weights between hidden-layer and output-layer obtained in the previous iteration. 
         [0045]    Substep 3.2, according to the output value of the samples on the output-layer neurons, the samples are clustered. On basis of which the number of the output-layer neurons is equal to the number of clusters, assigning rule is that a sample should be assigned to a cluster corresponding to the neuron with the highest output value of the output-layer neurons. 
         [0046]    For instance, when the number of clusters for Iris dataset of UCI is 3, output value of a sample on an output-layer neuron and clustering result are presented in Table 2: 
         [0000]    
       
         
               
             
               
               
               
               
             
               
               
               
               
             
           
               
                 TABLE 2 
               
             
             
               
                   
               
               
                 table for output value of a sample on an 
               
               
                 output-layer neuron and clustering result 
               
             
          
           
               
                 Sam- 
                   
                 Output value of a 
                   
               
               
                 ple 
                 Eigenvectors of the 
                 samples on an 
                 Cluster 
               
               
                 NO. 
                 sample x i   
                 output-layer neuron o i   
                 assigned 
               
               
                   
               
             
          
           
               
                 1 
                 [0.66, 0.41, 0.71, 0.91] 
                 [0.18, 0.90, −0.08] 
                 Second class 
               
               
                 2 
                 [0.61, 0.41, 0.76, 0.70] 
                 [0.31, 0.72, −0.03] 
                 Second class 
               
               
                 3 
                 [0.66, 0.54, 0.79, 0.83] 
                 [−0.15, 1.10, 0.05] 
                 Second class 
               
               
                 4 
                 [0.69, 0.33, 0.64, 0.54] 
                 [0.42, 0.52, 0.04] 
                 Second class 
               
               
                 5 
                 [0.63, 0.37, 0.61, 0.50] 
                 [0.36, 0.49, 0.13] 
                 Second class 
               
               
                 6 
                 [0.22, 0.70, 0.08, 0.12] 
                 [−0.26, 0.19, 1.07] 
                 Third class 
               
               
                 7 
                 [0.69, 0.50, 0.83, 0.91] 
                 [−0.06, 1.12, −0.05] 
                 Second class 
               
               
                 8 
                 [0.11, 0.50, 0.10, 0.04] 
                 [0.58, −0.37, 0.79] 
                 Third class 
               
               
                 9 
                 [0.44, 0.41, 0.54, 0.58] 
                 [0.47, 0.37, 0.14] 
                 First class 
               
               
                 10 
                 [0.66, 0.45, 0.76, 0.70] 
                 [0.22, 0.77, 0.01] 
                 Second class 
               
               
                 11 
                 [0.25, 0.29, 0.49, 0.54] 
                 [1.12, −0.11, −0.01] 
                 First class 
               
               
                 12 
                 [0.52, 0.08, 0.59, 0.58] 
                 [1.41, −0.11, −0.31] 
                 First class 
               
               
                 13 
                 [1.00, 0.75, 0.91, 0.79] 
                 [−1.21, 1.86, 0.35] 
                 Second class 
               
               
                 14 
                 [0.13, 0.58, 0.10, 0.04] 
                 [0.27, −0.18, 0.91] 
                 Third class 
               
               
                 15 
                 [0.25, 0.87, 0.08, 0.00] 
                 [−0.81, 0.45, 1.37] 
                 Third class 
               
               
                 . . . 
                 . . . 
                 . . . 
                 . . . 
               
               
                   
               
             
          
         
       
     
         [0047]    Next, on the basis of clustering result of step 3, task of step 4 is to select multiple samples from each cluster as exemplars for the cluster according to a rule. 
         [0048]    There are usually two kinds of rules: 1) choosing ρ samples which are the nearest to a cluster center in a cluster (that is a mean of all samples of the cluster) as exemplars of the cluster on original space, wherein ρ is a number of exemplars usually with value range of [5,10]; 2) choosing ρ samples which are the nearest to a cluster center in a cluster as exemplars for the cluster on hidden-layer space (also called extreme learning machine feature space). 
         [0049]    The present invention adopts the first kinds of rules, therefore step 4 can be divided into following two substeps: 
         [0050]    Substep 4.1, according to the clustering result of step 3, the present invention calculates each cluster mean MEAN. For instance, when the number of clusters for Iris dataset of UCI is 3, cluster means are presented in Table 3: 
         [0000]    
       
         
               
             
               
               
               
               
             
               
               
               
               
             
           
               
                 TABLE 3 
               
             
             
               
                   
               
               
                 table for cluster means 
               
             
          
           
               
                 Sam- 
                   
                   
                   
               
               
                 ple 
                 Eigenvector of the 
                 Cluster 
               
               
                 NO. 
                 samples x i   
                 assigned 
                 Cluster mean MEAN 
               
               
                   
               
             
          
           
               
                 9 
                 [0.44, 0.41, 0.54, 0.58] 
                 First class 
                 [0.40, 0.25, 0.52, 0.54] 
               
               
                 11 
                 [0.25, 0.29, 0.49, 0.54] 
                 First class 
               
               
                 12 
                 [0.52, 0.08, 0.59, 0.58] 
                 First class 
               
               
                 . . . 
                 . . . 
                 . . . 
               
               
                 1 
                 [0.66, 0.41, 0.71, 0.91] 
                 Second class 
                 [0.69, 0.47, 0.76, 0.70] 
               
               
                 2 
                 [0.61, 0.41, 0.76, 0.70] 
                 Second class 
               
               
                 3 
                 [0.66, 0.54, 0.79, 0.83] 
                 Second class 
               
               
                 4 
                 [0.69, 0.33, 0.64, 0.54] 
                 Second class 
               
               
                 5 
                 [0.63, 0.37, 0.61, 0.50] 
                 Second class 
               
               
                 13 
                 [1.00, 0.75, 0.91, 0.79] 
                 Second class 
               
               
                 7 
                 [0.69, 0.50, 0.83, 0.91] 
                 Second class 
               
               
                 10 
                 [0.66, 0.45, 0.76, 0.70] 
                 Second class 
               
               
                 . . . 
                 . . . 
                 . . . 
               
               
                 6 
                 [0.22, 0.70, 0.08, 0.12] 
                 Third class 
                 [0.16, 0.63, 0.09, 0.05] 
               
               
                 8 
                 [0.11, 0.50, 0.10, 0.04] 
                 Third class 
               
               
                 14 
                 [0.13, 0.58, 0.10, 0.04] 
                 Third class 
               
               
                 15 
                 [0.25, 0.87, 0,08, 0.00] 
                 Third class 
               
               
                 . . . 
                 . . . 
                 . . . 
               
               
                   
               
             
          
         
       
     
         [0051]    Substep 4.2, in each cluster, distances between each sample of the cluster and a cluster center (Euclidean distance) are calculated and ρ samples which are the nearest to the cluster center (cluster mean) are selected to form an exemplar set of current iteration E (m) . The distance between each sample of the cluster and the cluster center are presented in Table 4 and the exemplar sets of current iteration E (m)  are presented in Table 5: 
         [0000]    
       
         
               
             
               
               
               
               
             
               
               
               
               
             
           
               
                 TABLE 4 
               
             
             
               
                   
               
               
                 table for the distances between each sample 
               
               
                 of a cluster and a cluster center 
               
             
          
           
               
                   
                   
                   
                 distances between 
               
               
                 Sam- 
                   
                   
                 each sample of a 
               
               
                 ple 
                 Eigenvector of the 
                 Cluster 
                 cluster and a cluster 
               
               
                 NO. 
                 samples x i   
                 assigned 
                 center 
               
               
                   
               
             
          
           
               
                 9 
                 [0.44, 0.41, 0.54, 0.58] 
                 First class 
                 0.1709 
               
               
                 11 
                 [0.25, 0.29, 0.49, 0.54] 
                 First class 
                 0.1581 
               
               
                 12 
                 [0.52, 0.08, 0.59, 0.58] 
                 First class 
                 0.2232 
               
               
                 . . . 
                 . . . 
                 . . . 
                 . . . 
               
               
                 1 
                 [0.66, 0.41, 0.71, 0.91] 
                 Second class 
                 0.2361 
               
               
                 2 
                 [0.61, 0.41, 0.76, 0.70] 
                 Second class 
                 0.1000 
               
               
                 3 
                 [0.66, 0.54, 0.79, 0.83] 
                 Second class 
                 0.1536 
               
               
                 4 
                 [0.69, 0.33, 0.64, 0.54] 
                 Second class 
                 0.2441 
               
               
                 5 
                 [0.63, 0.37, 0.61, 0.50] 
                 Second class 
                 0.2759 
               
               
                 13 
                 [1.00, 0.75, 0.91, 0.79] 
                 Second class 
                 0.4529 
               
               
                 7 
                 [0.69, 0.50, 0.83, 0.91] 
                 Second class 
                 0.2234 
               
               
                 10 
                 [0.66, 0.45, 0.76, 0.70] 
                 Second class 
                 0.0361 
               
               
                 . . . 
                 . . . 
                 . . . 
                 . . . 
               
               
                 6 
                 [0.22, 0.70, 0.08, 0.12] 
                 Third class 
                 0.1162 
               
               
                 8 
                 [0.11, 0.50, 0.10, 0.04] 
                 Third class 
                 0.1400 
               
               
                 14 
                 [0.13, 0.58, 0.10, 0.04] 
                 Third class 
                 0.0600 
               
               
                 15 
                 [0.25, 0.87, 0.08, 0.00] 
                 Third class 
                 0.2613 
               
               
                 . . . 
                 . . . 
                 . . . 
                 . . . 
               
               
                   
               
             
          
         
       
     
         [0000]    
       
         
               
             
               
               
               
               
             
               
               
               
               
             
           
               
                 TABLE 5 
               
             
             
               
                   
               
               
                 table for the exemplar set of current iteration E (m)   
               
             
          
           
               
                   
                   
                   
                 The distances between 
               
               
                 Sam- 
                   
                   
                 each sample in the 
               
               
                 ple 
                 Eigenvector of the 
                 Cluster 
                 cluster and the cluster 
               
               
                 NO. 
                 samples x i   
                 assigned 
                 center 
               
               
                   
               
             
          
           
               
                 9 
                 [0.44, 0.41, 0.54, 0.58] 
                 First class 
                 0.1709 
               
               
                 11 
                 [0.25, 0.29, 0.49, 0.54] 
                 First class 
                 0.1581 
               
               
                 12 
                 [0.52, 0.08, 0.59, 0.58] 
                 First class 
                 0.2232 
               
               
                 2 
                 [0.61, 0.41, 0.76, 0.70] 
                 Second class 
                 0.1000 
               
               
                 3 
                 [0.66, 0.54, 0.79, 0.83] 
                 Second class 
                 0.1536 
               
               
                 10 
                 [0.66, 0.45, 0.76, 0.70] 
                 Second class 
                 0.0361 
               
               
                 6 
                 [0.22, 0.70, 0.08, 0.12] 
                 Third class 
                 0.1162 
               
               
                 8 
                 [0.11, 0.50, 0.10, 0.04] 
                 Third class 
                 0.1400 
               
               
                 14 
                 [0.13, 0.58, 0.10, 0.04] 
                 Third class 
                 0.0600 
               
               
                   
               
             
          
         
       
     
         [0052]    It is noted that: if the number of samples of a cluster is less than the number of exemplars ρ required to be selected from the cluster, all samples of the cluster are selected to be exemplars for the cluster; and if a cluster doesn&#39;t contain any sample, a sample as a exemplar for the cluster should be randomly selected from all samples with exception of exemplar sets for all the other clusters. 
         [0053]    Then, in step 5, the extreme learning machine is retrained by using the exemplar sets of each cluster obtained from step 4. Step 5 comprises two substeps: 
         [0054]    Substep 5.1, target functions are constructed for exemplar sets obtained from step 4, as indicated in Table 6: 
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 5 
               
             
             
               
                   
               
               
                 table for the exemplar sets and corresponding target output 
               
             
          
           
               
                 Exemplar set of current 
                 Corresponding target 
                 Classification of 
               
               
                 iteration E (m)   
                 output T (m)   
                 a sample 
               
               
                   
               
               
                 [0.44, 0.41, 0.54, 0.58] 
                 [1 0 0] 
                 First class 
               
               
                 [0.25, 0.29, 0.49, 0.54] 
                 [1 0 0] 
                 First class 
               
               
                 [0.52, 0.08, 0.59, 0.58] 
                 [1 0 0] 
                 First class 
               
               
                 [0.61, 0.41, 0.76, 0.70] 
                 [0 1 0] 
                 Second class 
               
               
                 [0.66, 0.54, 0.79, 0.83] 
                 [0 1 0] 
                 Second class 
               
               
                 [0.66, 0.45, 0.76, 0.70] 
                 [0 1 0] 
                 Second class 
               
               
                 [0.22, 0.70, 0.08, 0.12] 
                 [0 0 1] 
                 Third class 
               
               
                 [0.11, 0.50, 0.10, 0.04] 
                 [0 0 1] 
                 Third class 
               
               
                 [0.13, 0.58, 0.10, 0.04] 
                 [0 0 1] 
                 Third class 
               
               
                   
               
             
          
         
       
     
         [0055]    Substep 5.2, the present invention calculates an output of the hidden-layer neurons H (m)  and output weights between hidden-layer and output-layer β (m)  of the new extreme learning machine after being retrained through Eq. (7), Eq. (8) and Eq. (9): 
         [0000]    
       
         
           
             
               
                 
                   
                     H 
                     
                       ( 
                       m 
                       ) 
                     
                   
                   = 
                   
                     g 
                      
                     
                       ( 
                       
                         
                           W 
                           · 
                           
                             E 
                             
                               ( 
                               m 
                               ) 
                             
                           
                         
                         + 
                         
                           B 
                           
                             ( 
                             m 
                             ) 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       if 
                        
                       
                           
                       
                        
                       K 
                       × 
                       ρ 
                     
                     &gt; 
                     L 
                   
                   , 
                   
                     
                       β 
                       
                         ( 
                         m 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ( 
                           
                             
                               
                                 H 
                                 
                                   
                                     ( 
                                     m 
                                     ) 
                                   
                                   T 
                                 
                               
                                
                               
                                 H 
                                 
                                   ( 
                                   m 
                                   ) 
                                 
                               
                             
                             + 
                             
                               
                                 I 
                                 L 
                               
                               γ 
                             
                           
                           ) 
                         
                         † 
                       
                        
                       
                         H 
                         
                           
                             ( 
                             m 
                             ) 
                           
                           T 
                         
                       
                        
                       
                         T 
                         
                           ( 
                           m 
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
             
               
                 
                   
                     else 
                      
                     
                         
                     
                      
                     
                       β 
                       
                         ( 
                         m 
                         ) 
                       
                     
                   
                   = 
                   
                     
                       
                         
                           H 
                           
                             
                               ( 
                               m 
                               ) 
                             
                             T 
                           
                         
                          
                         
                           ( 
                           
                             
                               
                                 H 
                                 
                                   ( 
                                   m 
                                   ) 
                                 
                               
                                
                               
                                 H 
                                 
                                   
                                     ( 
                                     m 
                                     ) 
                                   
                                   T 
                                 
                               
                             
                             + 
                             
                               
                                 I 
                                 
                                   K 
                                   × 
                                   ρ 
                                 
                               
                               γ 
                             
                           
                           ) 
                         
                       
                       † 
                     
                      
                     
                       T 
                       
                         ( 
                         m 
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
         [0000]    wherein B (m)  is a row-expansive matrix of the bias b whose number of rows is equal to that of E (m)  which is a matrix with K×ρ rows and L columns, K being the number of clusters, ρ being the number of exemplars of each cluster, and L being the number of hidden-layer neurons, H (m)     T    represents a transpose of the matrix H (m) , (•) †  represents the generalized inverse matrix, I L  is the identity matrix of L dimension, and I K×ρ  is an identity matrix of K×ρ dimension. 
         [0056]    At last, the present invention sets termination conditions of the iteration and step 6 completes a judgment to the termination conditions. If any of the termination conditions is met, current clustering result is outputted, otherwise next iteration requires to progress by going back to step 3 with adding one to m that represents the number of iterations. 
         [0057]    There are two termination conditions adopted by the present invention. 
         [0058]    Once any of these two termination condition is met, the iteration ends. 
         [0059]    1) if the number of iterations is greater than a maximal limit of the number of iterations, which meets Eq.(10), the iteration ends: 
         [0000]        m &gt;MaxIteration  (10)
 
         [0000]    wherein m represents current number of iterations and MaxIteration is the maximal limit of the number of iterations. 
         [0060]    2) if a sum of squares of a cluster S is steady, the iteration ends. The sum of squares of a cluster S is a sum of squares of distances, each of distance being an distance between an observation point and corresponding cluster center. Equation for calculating the sum of squares of a cluster S is: 
         [0000]        s=Σ   k=1   K Σ xεC     k     ∥x −MEAN k ∥ 2    (11)
 
         [0000]    wherein C k  stands for current sample set of the kth cluster, x represents a sample, MEAN k  stands for the kth cluster mean. 
         [0061]    If following condition is met by consecutive five iterations, the sum of squares of a cluster S is justified to be steady: 
         [0000]        S   (m)   −S   (m-1)   &lt;ε,m≧ 2   (12)
 
         [0000]    wherein S (m)  represents a sum of squares of a cluster of current iteration, S (m-1)  represents a sum of squares of a cluster of previous iteration, and E stands for a very small positive number. 
         [0062]    Although the embodiments of the present invention have been disclosed above, but it is not limited to the applications set forth in the specification and embodiments, and can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications may be easily achieved without departing the general concept defined by the claims and their equivalent. The present invention is not limited to particular details and illustrations shown and described herein.