Influence of kernel clustering on an RBFN

CAAI Transactions on Intelligence TechnologyVolume 4, Issue 4 p. 255-260 ArticleOpen Access Influence of kernel clustering on an RBFN Changming Zhu, Changming Zhu orcid.org/0000-0001-8039-5601 College of Information Engineering, Shanghai Maritime University, No. 1550, Haigang Avenue, Shanghai, People's Republic of China Department of Computer Science and Technology, Tongji University, No. 4800, Cao'an Highway, Shanghai, People's Republic of ChinaSearch for more papers by this authorDuoqian Miao, Corresponding Author Duoqian Miao dqmiao@tongji.edu.cn Department of Computer Science and Technology, Tongji University, No. 4800, Cao'an Highway, Shanghai, People's Republic of ChinaSearch for more papers by this author Changming Zhu, Changming Zhu orcid.org/0000-0001-8039-5601 College of Information Engineering, Shanghai Maritime University, No. 1550, Haigang Avenue, Shanghai, People's Republic of China Department of Computer Science and Technology, Tongji University, No. 4800, Cao'an Highway, Shanghai, People's Republic of ChinaSearch for more papers by this authorDuoqian Miao, Corresponding Author Duoqian Miao dqmiao@tongji.edu.cn Department of Computer Science and Technology, Tongji University, No. 4800, Cao'an Highway, Shanghai, People's Republic of ChinaSearch for more papers by this author First published: 14 November 2019 https://doi.org/10.1049/trit.2019.0036Citations: 26AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Classical radial basis function network (RBFN) is widely used to process the non-linear separable data sets with the introduction of activation functions. However, the setting of parameters for activation functions is random and the distribution of patterns is not taken into account. To process this issue, some scholars introduce the kernel clustering into the RBFN so that the clustering results are related to the parameters about activation functions. On the base of the original kernel clustering, this study further discusses the influence of kernel clustering on an RBFN when the setting of kernel clustering is changing. The changing involves different kernel-clustering ways [bubble sort (BS) and escape nearest outlier (ENO)], multiple kernel-clustering criteria (static and dynamic) etc. Experimental results validate that with the consideration of distribution of patterns and the changes of setting of kernel clustering, the performance of an RBFN is improved and is more feasible for corresponding data sets. Moreover, though BS always costs more time than ENO, it still brings more feasible clustering results. Furthermore, dynamic criterion always cost much more time than static one, but kernel number derived from dynamic criterion is fewer than the one from static. 1 Introduction Neural network (NN), which is also called artificial NN (ANN) [[1], [2]], is a mathematical or computational model. There are many kinds of ANNs. In terms of learning algorithms, supervised or unsupervised learning networks [[3], [4]], hybrid learning networks [[5]], associate learning networks [[6]], and optimisation application networks [[7]] are used widely. In terms of network connectionism, feed-forward networks [[8]], recurrent networks [[9]], and reinforcement networks [[10]] are used widely. Regardless of which kind of network is used, the structure of a network always consists of three parts: input layer, hidden layer, and output layer. In the hidden layer, mapping functions are called activation functions. Kernel functions are always treated as activation functions. The most well known activation functions are radial basis functions (RBFs) which are also treated as RBF kernels. A network with RBF kernels is named an RBF network (RBFN). RBFN is a linear combination of RBF kernels. RBF kernels have two parameters, centres, and radiuses (or widths). They are determined by unsupervised or supervised learning [[11], [12]]. In RBFN, the connection weights between hidden units and output units and the thresholds of output units are adjusted by supervised learning rules. Similar to multi-layered perception (MLP), supervised learning rules of RBFN such as steepest descent method can be applied to minimise the sum of squared errors, which are the differences between the inference outputs of the model and the desired outputs. In unsupervised learning, the mean of patterns, determined with K -means algorithms, can be used as the centre of kernel, and then one can determine the appropriate size of the radius with the nearest neighbour rule [[12]]. The activation function of RBFN is given by the following equation: (1) where L is the number of hidden units and x is an input feature vector, namely, pattern. and are centre and radius of the j th RBF kernel , respectively. In the original RBFN, activation functions always have a same setting and without the consideration of data structure. This makes patterns in an L- dimensional space have no feasible distribution though they are linear separable. From work [[13]], Gao and Li proposed a new kernel nearest neighbour classification method, such that a pattern is labelled according to the minimum distance between itself and the surfaces of the existing kernels. We can adopt their kernel-generating method to gain more feasible kernels for RBFN. In fact, there are many factors that should be considered during the process of the generation of kernels. This paper proposes a new kind of kernel-clustering method. Our basic ideas differ from the one given in [[13]]. Here, different kernel criteria and ways, ways of the computation of kernel centre, and width should be taken into account. Finally, experimental results on Letter, Pen digit, Shuttle, US Postal Service (USPS), Iris, and Two Spiral demonstrate the effectiveness of the proposed method. 2 Related work 2.1 Original kernel clustering Gao and Li [[13]] propose a kernel nearest neighbour method, which aims to label a pattern according to the minimum distance between itself and the surfaces of the existing kernels. In [[13]], a kernel is treated as a cluster and the process is given below: (i) For a data set, we first choose a class as the object class and select a pattern as the initial centre for a kernel. Here, the class being clustered is called the object class. Corresponding patterns of this class are called object patterns. Other classes are called non-object classes and the corresponding patterns are called outliers or non-object patterns. (ii) We compute the minimum Euclid distance between patterns from the object class and non-object class and the centre of the present object kernel and denote the two distances as and . The corresponding patterns are denoted as and . (iii) If , put into this kernel and then update the kernel. Otherwise, stop growing the kernel and choose the next position to generate a new object kernel. In [[13]], for the updated kernel, we should first update the centre. The new centre is the mean of new object patterns in this kernel. Then, compute the new and . The updated width is the mean of new and ; specifically, . (iv) Repeat steps (ii) and (iii), we cover class by kernels, until all patterns from the training subsets in this object class are covered by different kernels. In our method, we use this kernel-clustering method as the basic algorithm and provide some new ideas to improve it. 2.2 Computation about kernel centre and radius of a Gaussian unit RBF is always defined as a monotonic function of the Euler distance from point x in the input space into a centre and it can be denoted as ; the function is local: namely, when x is far from , the value is small. The most widely used RBF is written as , where is the centre of a kernel, is its width and it controls the area of radius function. In RBFN, these two parameters can be changed. Some methods are widely used: (i) To select the centre based on experience. For example, as long as the distribution of training patterns can represent a given problem, we can select M centres based on experience of the uniform distribution, and the spacing is d, the width of the Gaussian kernel function is . (ii) To select the basis function by a clustering method. We can put the cluster centres as the centres of basis functions, with the variance of the various types of patterns as the width parameters of each basis function. After determining the parameters of the hidden layer of basis functions, as the output layer is a linear unit, its weight can simply be calculated directly using the least-squares method. (iii) The centre, width, and weights can be calculated by an error-correction algorithm through the training patterns. The practise is similar with BP, which is used to calculate the derivatives of the error parameters of parameters, and then use iteration to determine the parameters. We can fix centres and widths, only to learn the weight values. 3 New kernel clustering Here, we change the setting of the original kernel clustering and develop a new kernel clustering. The changes involve different ways for kernel clustering and their criteria, and different computation ways about kernel centres and widths. Then, we apply newly generated kernels into RBFN. The basic steps of this new method are similar with the original one, namely, choose the initial centre, compute the distance, cover the object patterns, and update the kernels. The differences are that in our method, we use different kernel-clustering ways and criteria to generate kernels. The widths of kernels have other equations for computation. The algorithm will stop until all object patterns in each class have been covered by different kernels. The framework of our work is given in Fig. 1. From this table, it is found that the algorithm has at least three parameters and in terms of RBFN, different parameters will bring different performances. Fig. 1Open in figure viewerPowerPoint Algorithm: kernel clustering 3.1 Computation of kernel centre and width Here, denotes the furthest distance between patterns in the object class and the centre of the present kernel while means the nearest distance between patterns from non-object classes and the centre of the present kernel. Then, we adopt the following five ways to determine the width of kernel. Equation (4) gives the method, which is used in the original way. Fig. 2 shows the differences between these five ways. From the outer to the inner are five width contours with (2)–(6). From the contours, we find that (2) and (5) make kernel tight, (3) makes it somewhat loose, and (4) is moderate while (6) is so small as not to cover all object patterns. Furthermore, (6) can decrease the chance of misclassifying testing patterns because the width radius is diminished (2) (3) (4) (5) (6) Fig. 2Open in figure viewerPowerPoint Differences between five ways to compute the width of the kernel 3.2 Kernel-clustering method Two kinds of kernel-clustering ways are adopted here. One is bubble sort (BS) and the other is escape nearest outlier (ENO). BS aims to cover object patterns one by one. In BS, it computes the minimum distance between patterns from object class and non-object class and the centre of the present object kernel Then it compares the two distances so as to decide whether to include this new object pattern into present kernel or not. If yes, cover the new pattern and update the kernel; otherwise, stop growing the present kernel and choose a next position to generate a new object kernel. ENO aims to generate one object kernel by computing the nearest distance between patterns from non-object classes and the centre of the present kernel. This distance is denoted as . Then, ENO puts all object patterns whose distance to the centre is less than into the object kernel. The centre of this kernel is not changed and the width can be updated with (2)–(6). BS generates kernels well, but it costs too much time to cover object patterns one by one. ENO contrasts with BS in terms of these performances. However, we should note that the number of kernels with BS or ENO has no absolute relationship. 3.3 Kernel-clustering criteria Two kinds of kernel-clustering criteria are adopted in our work. One is static criterion and the other is dynamic criterion. The static kernel clustering is similar with the original kernel clustering (please see Section 2.1) and the only difference is the different way to compute kernel widths. Static kernel clustering adopts (2)–(6) to compute the kernel widths while the original kernel clustering adopts ways given in Section 2.2 to compute kernel widths. Compared to the static kernel clustering, dynamic one is more complicated and its steps are given in Fig. 3. In this table, indicates the i th pattern and is the j th kernel. is 'left index' and it indicates the total number of patterns which are only covered by . Fig. 3Open in figure viewerPowerPoint Algorithm: dynamic kernel clustering From steps of these two criteria, it is found that dynamic criterion is more time-consuming. However, with dynamic criterion, the number of kernels is fewer since in dynamic kernel clustering, we choose a kernel, which can cover maximum patterns uniquely at each time. 4 Experiment To validate the effectiveness of our work, we adopt data sets Letter, Pen digit, Shuttle, USPS, Iris, and Two Spiral for experiments, and then we provide further discussion in terms of (i) differences between BS and ENO, (ii) differences between static and dynamic criteria, (iii) convergence analysis, (iv) generalisation risk analysis, and (v) the testing results of RBFN with different kernel-clustering ways and the comparison between our method and others. 4.1 Experimental setting The experiment setting is shown in Table 1. Main numerical characteristics are summarised in Table 2. Here, columns A–M denote data set, number of training set, number of testing set, number of classes, number of attributes, the maximum data of training set, the minimum data of training set, the mean of training set, the co-variance of training set, the maximum data of testing set, the minimum data of testing set, the mean of testing set, and the co-variance of testing set. Our experimental environment is simple and crude. All computations are performed on Pentium D processor with 2.66 GHz, 512 M RAM, Microsoft Windows XP, and MATLAB 7.0 environment. This configuration leads to somewhat long computations. However, this does not influence conclusions derived from the experiments. Table 1. Experimental setting Experimental item Value kernel-clustering way (a) BS, (b) ENO kernel-clustering criterion (a) static, (b) dynamic kernel centre computing mean of patterns kernel width computing (2)–(6) BS: bubble sort and ENO: escape nearest outlier. Table 2. Basic characteristics of Letter, Pen Digit, Shuttle, USPS, Iris, PID, Two Spiral, and 2D-VOW data sets A B C D E F G Letter 16,000 4000 26 16 15 0 Pen Digit 7494 3498 10 16 100 0 Two Spiral 194 194 2 2 6.81 −6.81 Shuttle 43,500 14,500 7 9 13,148 −13,839 USPS 7291 2007 10 256 0.99 −1 Iris 75 75 3 4 7.7 0.1 A H I J K L M Letter 5.92 6.09 15 0 5.93 6.3 Pen Digit 50.70 1772.29 100 0 50.73 1727.55 Two Spiral 8.54 6.81 −6.81 8.78 Shuttle 30.16 920.03 15,164 −26,739 30.37 924.17 USPS −0.51 0.24 1 −1 −0.48 0.27 Iris 3.49 5.769 7.9 0.1 3.44 5.66 Columns A–M denote data set, number of training set, number of testing set, number of classes, number of attributes, the maximum data of training set, the minimum data of training set, the mean of training set, the co-variance of training set, the maximum data of testing set, the minimum data of testing set, the mean of testing set, and the co-variance of testing set, respectively. USPS: US Postal Service. 4.2 Kernel-clustering results Here, we use Letter, Pen digit, Shuttle, USPS, Iris, and Two Spiral to show the effectiveness of the proposed kernel-clustering method. We adopt different kernel-clustering ways, criteria, and ways to compute kernel centres and widths so as to get kernel-clustering results. Table 3 gives the combinations of these different parameters. Table 4 gives the corresponding kernel results. From the results, we find that the results are better than those in [[13]] in average. It is found that BS always costs much more time than ENO, but BS generates more feasible kernels. What is more, dynamic criterion always cost much more time than static one, but with dynamic criterion, the kernel number is smaller than the one with static criterion. Table 3. Parameter configurations Parameters Clustering Criterion Width and centre P1 BS dynamic (2) P2 BS dynamic (3) P3 BS dynamic (4) P4 BS dynamic (5) P5 BS dynamic (6) P6 ENO dynamic (2) P7 ENO dynamic (3) P8 ENO dynamic (4) P9 ENO dynamic (5) P10 ENO dynamic (6) P11 BS static (2) P12 BS static (3) P13 BS static (4) P14 BS static (5) P15 BS static (6) P16 ENO static (2) P17 ENO static (3) P18 ENO static (4) P19 ENO static (5) P20 ENO static (6) BS: bubble sort and ENO: escape nearest outlier. Table 4. Kernel-clustering results Parameters Letter Pen Digit Shuttle USPS Iris Two spiral A, % B, % A, % B, % A, % B, % A, % B, % A, % B, % A, % B, % P1 8.46 2.67 0.65 5.17 10.67 29.9 P2 8.41 2.66 0.63 5.28 10.71 28.47 P3 8.95 2.68 0.61 5.57 10.68 28.95 P4 8.94 2.73 0.59 5.49 10.48 28.12 P5 8.83 2.78 0.54 5.38 11.17 26.63 P6 9.02 2.89 0.55 5.53 10.62 27.4 P7 9.67 2.9 0.51 5.65 10.44 29.43 P8 10.2 2.75 0.51 5.9 11.2 29.29 P9 9.99 2.9 0.53 5.73 11.19 30.68 P10 9.97 2.93 0.53 5.92 10.54 30.1 P11 12.3 3.75 0.64 7.56 11.7 39.23 P12 10.85 3.64 0.63 6.39 13.62 39.49 P13 11.39 3.73 0.69 7.24 12.4 38.67 P14 11.38 3.58 0.66 7.48 11.68 32.22 P15 9.77 3.57 0.73 6.12 13.69 34.45 P16 12.09 3.7 0.77 7.62 14.49 34.47 P17 12.83 3.62 0.81 7.25 10.71 31.21 P18 10.73 3.01 0.84 7.53 12.8 36.99 P19 11.06 3.22 0.67 7.07 11.34 37.59 P20 10.91 3.55 0.84 5.91 12.45 37.71 Gao and Li [[13]] 12.85 3.99 0.87 7.87 14.22 39.93 In each cell, A means (the number of kernels)/(number of training patterns) × 100 (%) and B means time for kernel formation (s). USPS: US Postal Service. 4.3 Further discussion We also give some further discussion in terms of the following five parts: (i) The difference between BS and ENO: In fact, in the previous experiments, we can find that BS can generate more feasible kernels than ENO does, while it costs much more time compared with ENO. (ii) The difference between static and dynamic criterion: Static criterion saves time for kernel clustering while dynamic criterion can generate fewer kernels. (iii) For assessing the effectiveness of a classifier, two criteria should be adopted. One is convergence and the other is generalisation risk. We adopt these two criteria so as to validate the effectiveness of the proposed method. (iv) Testing results of RBFN with different kernel-clustering ways: From the results, we find that for most cases, results under the new RBF are better than those without RBF. Comparisons between our method and other methods, for example, Naïve Bayes [[14], [15]], AdaBoost [[16]] (the standard ensemble algorithm), C4.5 [[17], [18]] (the most popular decision tree method), K -means method [[19]] (for determining the hidden structure directly), the k -nearest neighbour algorithm (KNN) [[19]], MLP [[20]] (a method that uses back propagation to estimate RBFNN), and probabilistic NN (PNN) [[21]] are also given. 4.3.1 Difference between BS and ENO In our proposed method, we use two kernel-clustering ways: BS and ENO. In terms of some data sets, Table 5 gives time of kernel clustering with these two methods. First, for USPS, Letter, and Two Spiral, we use static criterion, and in terms of the computation of width, we adopt (2). For Pen digit, we use static criterion, and adopt (5) as the way to compute the width. For Iris, static criterion and (6) are adopted. In terms of Shuttle, we use ENO, dynamic criterion, and (2) is used to compute the widths and centres of kernels. Table 5. Time of kernel clustering for BS and ENO Data set Shuttle USPS Pen Digit time for BS, s 298,333 37217.7 1564.08 kernel for BS 285 799 406 time for ENO, s 69,515 4885.37 276.7 kernel for ENO 292 911 407 time rate BS/ENO 4.29 7.62 5.65 kernel rate BS/ENO 0.98 0.88 1 Data set Letter Two Spiral Iris time for BS, s 315304.8 2.58 2.31 kernel for BS 1362 58 8 time for ENO, s 1962.98 0.75 0.49 kernel for ENO 3112 58 7 time rate BS/ENO 160.63 3.44 4.74 kernel rate BS/ENO 0.44 1 1.14 BS: bubble sort, ENO: escape nearest outlier, and USPS: US Postal Service. Time of kernel clustering for BS and ENO and corresponding numbers of kernels are shown in Table 5. From this table, we find that ENO can save more time than BS. However, BS generates fewer kernels than ENO does. The reason is that in terms of BS, the centre of a kernel is changed when a new object pattern comes. It should update the centre and width of this kernel each time when the new object pattern has been covered. However, in the process of ENO, the centre will not be changed and we regard the initial pattern as the kernel centre. 4.3.2 Difference between static and dynamic criteria Table 6 gives the best numbers of kernels with different kernel parameters and data sets. From this table, we find that though dynamic criterion costs more clustering time than static one, with dynamic one, the number of kernels is not more than or fewer than one with static criterion. From the principles of the two criteria, we know that dynamic criterion aims to choose kernels whose 'left indexes' are large enough to cover all object patterns. This is a 'positive' way. However, static criterion only uses the way, which is similar with the original kernel clustering to generate kernels. In contrast, dynamic one is more time-consuming than static one, because under dynamic one, we should make each pattern of object class be an initial centre of a kernel and compute the 'left index'. Table 6. Time of kernel clustering for static and dynamic criteria Data set Iris Letter Two Spiral time for static, s 0.14 267103.3 2.58 kernel for static 9 1354 58 time for dynamic, s 2.99 311189.5 10.59 kernel for dynamic 7 1354 58 Data set Shuttle USPS Pen Digit time for static, s 29437.31 7180.09 4333.2 kernel for static 391 789 304 time for dynamic, s 298333.29 205276.11 135685.89 kernel for dynamic 285 377 175 USPS: US Postal Service. 4.3.3 Convergence analysis For assessing the effectiveness of a classifier, two criteria should be adopted. One is convergence and the other is generalisation risk. Here, we adopt an empirical justification given in [[22]], so as to demonstrate that all compared methods can converge within limited iterations. Table 7 shows the numbers of iterations for these methods on all used data sets. From this table, it is found that the proposed method can converge in fewer iterations. This also means that the method has a simple structure. Table 7. Numbers of iterations comparisons for all compared methods on all data sets Data set Iris Letter Two Spiral Shuttle USPS Pen Digit RBFN (new best) 24 45 3 32 21 18 RBFN (original) 24 46 4 33 21 17 Bayes [[14], [15]] 24 46 4 33 22 19 AdaBoost [[16]] 25 48 4 35 20 19 C4.5 [[17], [18]] 30 53 4 41 24 23 k -means [[19]] 26 59 5 38 23 18 KNN [[19]] 30 57 4 41 26 24 MLP [[20]] 28 50 5 31 20 24 PNN [[21]] 34 48 4 36 30 22 RBFN: radial basis function network, KNN: k -nearest neighbour algorithm, MLP: multi-layered perception, PNN: probabilistic NN, and USPS: US Postal Service. 4.3.4 Generalisation risk analysis As one of the criteria to assess effectiveness of a method, the analysis of generalisation risk is important for interpreting the performance behaviour of a method. If the generalisation risk of a method is smaller, this method has a lower prediction error on label-unknown patterns. In [[23]], some researchers have interpreted it and given an algorithm about how to compute the generalisation risk of a method in statistics rather than compute the value of the objective function simply. Here, we still use the same way to compute the generalisation risk of the proposed method. Table 8 shows generalisation risks for all compared methods. From this table, in terms of the average generalisation risk, it is found that the proposed method has a smaller generalisation risk than others in average. This also means that the proposed method has a better prediction ability to label the testing patterns. Table 8. Generalisation risk comparison Data set Iris Letter Two Spiral Shuttle USPS Pen Digit RBFN (new best) 0.1231 0.0391 0.1232 0.2122 0.1893 0.0921 RBFN (original) 0.1243 0.0407 0.124 0.2131 0.1901 0.0985 Bayes [[14], [15]], 0.124 0.0396 0.1278 0.2164 0.1921 0.1002 AdaBoost [[16]] 0.1289 0.0429 0.1286 0.2278 0.1948 0.1019 C4.5 [[17], [18]] 0.1496 0.0507 0.1385 0.2455 0.2295 0.1014 k -means [[19]] 0.1524 0.0412 0.1382 0.2835 0.2135 0.1129 KNN [[19]] 0.1367 0.0512 0.148 0.2519 0.2204 0.1131 MLP [[20]] 0.1273 0.0445 0.1449 0.2267 0.2032 0.095 PNN [[21]] 0.1739 0.0495 0.1445 0.2457 0.2195 0.1156 Smaller the value, the lower the generalisation risk. RBFN: radial basis function network, KNN: k -nearest neighbour algorithm, MLP: multi-layered perception, PNN: probabilistic NN, and USPS: US Postal Service. 4.3.5 Testing results of RBFN with different kernel-clustering ways and the comparison between our method and others Finally, we give the results of RBFN with our proposed method and the original way in Table 9. The average classification performances of our method and other methods are also given in this table. Table 9. Average classification performances of our method and other methods Data set Iris Letter Two Spiral Shuttle USPS Pen Digit RBFN (ours) 100 97.55 100 99.95 99.1l 98.86 RBFN (original) 98.67 95.68 97.98 99.94 94.11 98.91 Bayes [[14], [15]], 95.8 65.27 86.82 99.72 87.1 85.78 AdaBoost [[16]] 96.6 85.18 93.5 97 93.42 94.94 C4.5 [[17], [18]] 95.2 88.01 94.21 99.91 86.4 99.4 k -means [[19]] 83.33 72.22 91.16 99.37 91.59 78.42 KNN [[19]] 92 95 96.35 99.56 95.71 99.43 MLP [[20]] 94.28 86.83 94.18 99.18 94.26 96.24 PNN [[21]] 99.1 96.17 97.57 99.38 94.67 98.33 Best result is indicated in bold for each data set. RBFN: radial basis function network, KNN: k -nearest neighbour algorithm, MLP: multi-layered perception, PNN: probabilistic NN, and USPS: US Postal Service. Then, with a data set used, the process of selecting the best parameters of a compared method includes the following four steps: (i) except for the constant parameters, the methods used here have some adjustable parameters and different values of them affect the experimental results. Then, for each method, according to its corresponding parameters, we choose different initial values of them so as to generate all combinations of the corresponding parameters; (ii) for a combination of each method's corresponding parameters, we use one group of training patterns to train a temporary model and then use this trained model to classify the corresponding testing patterns so as to get the corresponding classification accuracy; (iii) under the case of this combination of the parameters, the process in the previous step is repeated ten times according to the corresponding ten groups of training and testing sets and ten corresponding classification performances are gotten. Then, we obtain the average classification accuracy, which is regarded as the experimental result of the present combination of the parameters; and (iv) for a method, after implementing all possible combinations of its corresponding parameters, we get their corresponding average classification performances. Among the performances, we choose the best one and the corresponding parameters are regarded as the best ones. Then, the corresponding average classification accuracy with the best parameters is regarded as the final experimental result for the classification problem of the data set with the method used. Then, for the classification problem of a data set, as we used many compared methods, we repeat the process for each one until we set the best parameters for all methods used here. As a result, we can also get the final classification performances of them for the classification problem. From these results, we find that after the process of our method, RBFN with the new generated kernels has a better performance under most cases. In fact, new kernels bring better activation functions to map the patterns into a kernel space. 5 Conclusion In this paper, we propose a new RBFN, which is based on a new kernel-clustering method. We consider many fields of kernel clustering and set different kinds of parameters about kernel clustering. Experiments have validated the influence of the parameters on the classification accuracy and time for clustering. In our work, we give some further discussion about (i) differences between BS and ENO, (ii) differences between static and dynamic criteria, (iii) convergence analysis, (iv) generalisation risk analysis, and (v) testing results of RBFN with different kernel-clustering ways and the comparison between our method and others. From these results, it is found that with kernel clustering, the classification performance becomes better than the original algorithm. Different parameters and ways bring different classification results. The most important matters derived from our experiments are (i) BS always costs more time than ENO, but BS has feasible kernels, (ii) dynamic criterion always costs more time than static one, but dynamic one brings fewer kernels. For our proposed method, we should learn that to achieve a better performance for classification, kernel clustering is necessary. In kernel clustering, different clustering parameters lead to different kernel results. Then, BS and dynamic criterion are the first choice for kernel clustering when the data set is not very large. If we process a large-scale data set, static criterion and ENO is a good choice. We should know, in terms of classification that BS and ENO will not bring a great difference. However, if we want to have more feasible kernels, BS is better than ENO. If we want to have fewer kernels, dynamic criterion is superior to static one. ENO and static criterion save time versus BS and dynamic one. In the future, we will find a good method that can save more time for clustering and bring fewer and more feasible kernels. We will also use more large-scale data sets in future experiments. 6 Acknowledgments This work was sponsored by the 'Chenguang Program' supported by the Shanghai Education Development Foundation and Shanghai Municipal Education Commission under Grant no. 18CG54. Furthermore, this work was also supported by the National Natural Science Foundation of China (CN) under Grant nos. 61602296 and 61673301; the Natural Science Foundation of Shanghai (CN) under Grant no. 16ZR1414500; Project funded by the China Postdoctoral Science Foundation under Grant no. 2019M651576; the National Key R&D Program of China (Grant no. 213), and the authors thank their supports. 7 References [1]Yang Z.B. Lin D.K.J. Zhang A.J.: 'Interval-valued data prediction via regularized artificial neural network', Neurocomputing, 2019, 331, pp. 336– 345 (doi: 10.1016/j.neucom.2018.11.063) [2]Lu L. Yu Y. Yang X.M. et al.: 'Time delay Chebyshev functional link artificial neural network', Neurocomputing, 2019, 329, pp. 153– 164 (doi: 10.1016/j.neucom.2018.10.051) [3]Liu J. Gong M.G. He H.B.: 'Deep associative neural network for associative memory based on unsupervised representation learning', Neural Netw., 2019, 113, pp. 41– 53 (doi: 10.1016/j.neunet.2019.01.004) [4]Oh C. Ham B. Kim H. et al.: 'OCEAN: object-centric arranging network for self-supervised visual representations learning', Expert Syst. Appl., 2019, 125, pp. 281– 292 (doi: 10.1016/j.eswa.2019.01.073) [5]Adhikari S.P. Yang C.J. Slot K. et al.: 'Hybrid no-propagation learning for multilayer neural networks', Neurocomputing, 2018, 321, pp. 28– 35 (doi: 10.1016/j.neucom.2018.08.034) [6]Stahlbuhk T. Shrader B. Modiano E.: 'Learning algorithms for scheduling in wireless networks with unknown channel statistics', Ad Hoc Netw., 2019, 85, pp. 131– 144 (doi: 10.1016/j.adhoc.2018.10.006) [7]Jin L. Li S. Hu B.: 'A survey on projection neural networks and their applications', Appl. Soft Comput., 2019, 76, pp. 533– 544 (doi: 10.1016/j.asoc.2019.01.002) [8]Yang J. Ma J.: 'Feed-forward neural network training using sparse representation', Expert Syst. Appl., 2019, 116, pp. 255– 264 (doi: 10.1016/j.eswa.2018.08.038) [9]Zia T.: 'Hierarchical recurrent highway networks', Pattern Recognit. Lett., 2019, 119, pp. 71– 76 (doi: 10.1016/j.patrec.2018.06.023) [10]Yao H.P. Chen X. Li M.Z. et al.: 'A novel reinforcement learning algorithm for virtual network embedding', Neurocomputing, 2018, 284, pp. 1– 9 (doi: 10.1016/j.neucom.2018.01.025) [11]Lĺćzaro M. Hayes M. H. Figueiras-Vidal A.R.: 'Training neural network classifiers through Bayes risk minimization applying unidimensional Parzen windows', Pattern Recognit., 2018, 77, pp. 204– 215 (doi: 10.1016/j.patcog.2017.12.018) [12]Gadaleta M. Rossi M.: 'IDNet: smartphone-based gait recognition with convolutional neural networks', Pattern Recognit., 2018, 74, pp. 25– 37 (doi: 10.1016/j.patcog.2017.09.005) [13]Gao D.Q. Li J.: 'Kernel fisher discriminants and kernel nearest neighbor methods: a comparative study for large-scale learning problems'. Int. Joint Conf. Neural Networks, 2006, pp. 1333– 1338 [14]Jiang L.X. Zhang L.G. Yu L.J. et al.: 'Class-specific attribute weighted Naïve Bayes', Pattern Recognit., 2019, 88, pp. 321– 330 (doi: 10.1016/j.patcog.2018.11.032) [15]Harzevili N.S. Alizadeh S.H.: 'Mixture of latent multinomial naive Bayes classifier', Appl. Soft Comput., 2018, 69, pp. 516– 527 (doi: 10.1016/j.asoc.2018.04.020) [16]Lee W. Jun C.H. Lee J.S.: 'Instance categorization by support vector machines to adjust weights in AdaBoost for imbalanced data classification', Inf. Sci., 2017, 381, pp. 92– 103 (doi: 10.1016/j.ins.2016.11.014) [17]Han L. Li W.J. Su Z.: 'An assertive reasoning method for emergency response management based on knowledge elements C4.5 decision tree', Expert Syst. Appl., 2019, 122, pp. 65– 74 (doi: 10.1016/j.eswa.2018.12.042) [18]Kudla P. Pawlak T.P.: 'One-class synthesis of constraints for mixed-integer linear programming with C4.5 decision trees', Appl. Soft Comput., 2018, 68, pp. 1– 12 (doi: 10.1016/j.asoc.2018.03.025) [19]Ismkhan H.: 'I –k -means? +: an iterative clustering algorithm based on an enhanced version of the k -means', Pattern Recognit., 2018, 79, pp. 402– 413 (doi: 10.1016/j.patcog.2018.02.015) [20]Mondal A. Ghosh A. Ghosh S.: 'Scaled and oriented object tracking using ensemble of multilayer perceptrons', Appl. Soft Comput., 2018, 73, pp. 1081– 1094 (doi: 10.1016/j.asoc.2018.09.028) [21]Kusy M. Kowalski P.A.: 'Weighted probabilistic neural network', Inf. Sci., 2018, 430–431, pp. 65– 76 (doi: 10.1016/j.ins.2017.11.036) [22]Ye J.: 'Generalized low rank approximations of matrices', Mach. Learn., 2005, 61, (1–3), pp. 167– 191 (doi: 10.1007/s10994-005-3561-6) [23]Scholkopf B. Taylor J.S. Smola A. et al.: ' Generalization bounds via eigenvalues of the gram matrix'. Technical Report 1999–035, 1999, NeuroColt Citing Literature Volume4, Issue4Special Issue: Rough Sets and Data MiningDecember 2019Pages 255-260 FiguresReferencesRelatedInformation