# An Optimized Neural Network Classification Method Based on Kernel Holistic Learning and Division

Authors: Hui Wen; Tongbin Li; Deli Chen; Jianlu Yang; Yan Che
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

An optimized neural network classification method based on kernel holistic learning and division (KHLD) is presented. The proposed method is based on the learned radial basis function (RBF) kernel as the research object. The kernel proposed here can be considered a subspace region consisting of the same pattern category in the training sample space. By extending the region of the sample space of the original instances, relevant information between instances can be obtained from the subspace, and the classifier’s boundary can be far from the original instances; thus, the robustness and generalization performance of the classifier are enhanced. In concrete implementation, a new pattern vector is generated within each RBF kernel according to the instance optimization and screening method to characterize KHLD. Experiments on artificial datasets and several UCI benchmark datasets show the effectiveness of our method.


---

## Body

## 1. Introduction

In the field of pattern recognition, set classification [1–3] is a common classification task. It is widely applied in text classification, speech recognition, image recognition, and multiple other fields. Taking the classification task based on image set as an example, each image set is composed of a class of image frames with a certain number of similar features. Due to the use of relevant information from adjacent frames, image changes can be effectively explored in the actual conditions. The main challenge is how to effectively integrate the information from all existing images to reach a reliable decision. A typical approach is to establish an optimized representation of different subsets of images and to achieve effective measurements between different subsets.Different from the set classification methods mentioned above, almost all the current neural network [4–7] optimization algorithms and models are based on the training and classification of instances instead of learning and partitioning the subspace region containing those instances. Because the classification surface of the network classifier is essentially determined by the probability distribution of the training samples, if the size of the training sample set is too small or the dimension of the classified dataset is too high, the error in the final classification will be relatively large, which leads to the reduction in the generalization performance of the neural network classifier.To improve this problem effectively, inspired by the idea of set classification, this paper attempts to introduce the idea of set classification into the neural network and presents an optimized neural network classification method based on kernel holistic learning and division (KHLD), which can improve the performance of the neural network classifier under a given sample set. Different from set classification, the method of KHLD is based on the effective coverage of a local region of the sample space, so the kernel proposed here can be considered a subspace region consisting of the same pattern category in the training sample space. Though it might not obtain the spatial distribution directly, relevant information between instances can be obtained from the subspace. The main reason is that the instances of the same pattern category are relatively close to each other in the spatial distribution and can be considered to have some similarity. Compared to single-pattern vector classification, KHLD considers the similarity information of the local region in the sample space. Due to the expansion of the region presented by the original pattern vector in the sample space, it can be improved to a certain extent when the size of the sample set is too small or when the dimension of the sample space is too high. On the other hand, KHLD can make the classifier’s boundary farther away from that of the original sample, which can further strengthen the robustness and generalization ability of the classifier.The primary task of achieving KHLD is the establishment and representation of the kernel. In this paper, considering the local characteristics of a certain region covered by the sample space, we take the Gaussian distribution function under different parameters as the representative to establish the corresponding subspace set. Moreover, to integrate the subkernel with different parameters and the mapping effect into the original sample space, we first construct the corresponding RBF kernel by learning the original sample space to realize the local mapping of the different regions of the sample space. Then, RBF kernels with different parameters are further studied and divided. Thus, the KHLD presented in this work has two meanings: the establishment of different RBF kernel parameters and the holistic division of the coverage region.Typical optimization algorithms for establishing RBF kernels with different parameters include K-mean clustering [8], fuzzy clustering [9, 10], orthogonal forward selection [11], evolutionary algorithm [12], particle swarm optimization [13], and other algorithms [14–16]. It is worth noting that the above methods for optimizing the RBF kernel parameters effectively combine the holistic information of the training sample space, but the number of hidden nodes in the RBF network cannot be determined automatically, which may lead to poor adaptability for different sample sets. To automatically estimate the number of RBF kernel parameters, several sequence learning RBF network kernel parameters, including minimum resource allocation network (MRAN) [17], sequential learning algorithm for growing and pruning the RBF (GAP-RBF) [18], and other incremental design of radial basis function networks [19–21], can be used. However, the holistic information of the sample space is not taken into account in these methods, and the classification performance will be affected to some extent.To generate the optimal number and parameters of the RBF kernel, in our previous work, an incremental learning algorithm for the hybrid RBF-BP network (ILRBF-BP) [22] and a hybrid structure adaptive RBF-ELM network (HSARBF-ELM) [23] are presented. In ILRBF-BP, the method of potential density clustering is presented to generate RBF kernels automatically, which utilizes the global distribution information of sample space. However, the local adaptability of each RBF kernel parameter in ILRBF-BP is not fully considered; this disadvantage is overcome by HSARBF-ELM. By combining the potential density clustering and the center-oriented heterogeneous sample repulsive force, the density information of different regions of the sample space and the neighborhood information of the region covered by the initial hidden nodes of the RBF can be used effectively. The optimal number and parameters of the RBF kernel can be generated adaptively according to the distribution of the sample space. However, when the size of the training sample set is too small or the dimension of the sample set is too high, the distribution of the sample set will be very sparse, which leads to the failure of the optimization algorithm to some extent, and the generalization performance of the neural network classifier will be reduced. To solve this problem, an optimal neural network based on KHLD is proposed. The premise of the method of KHLD is to establish the optimized kernel parameters. The geometry of these kernels is a regular hypersphere, and the optimization of the number and parameters of RBF kernels in HSARBF-ELM is just in line with this requirement. Thus, the RBF kernel established in HSARBF-ELM is the research object of this study.When the number and parameters of the optimized RBF kernel are established, the subsequent task is to realize KHLD. In practice, the training of the weights of network classifiers is carried out in a single instance. When all the RBF kernels are established, according to the probability density distribution of the pattern vectors in each subkernel, we consider generating new pattern vectors within each RBF kernel, which is equivalent to extending the existing pattern vector subset in the current RBF kernel, to characterize KHLD. Intuitively, when the number of samples generated in the region covered by the kernel is sufficient, the covered region can be approximated. In this way, the KHLD is transformed for training and dividing more pattern vectors. On the basis of generating a suitable sample set size, the existing network classifier is used for training and classification; thus, the final classification surface can be modified to improve the generalization performance of the network classifier.To achieve the effective expansion of the pattern vector in the region covered by the RBF kernel, a suitable sample probability distribution model is first needed to generate new pattern vectors. For this problem, we consider that the effective region covered by the RBF subkernel contains a certain number of original pattern vectors. In the region near the center, the probability density is relatively dense, and the probability density near the boundary is relatively sparse; thus, it can be considered that these pattern vectors similarly obey the multivariate Gaussian distribution with the current RBF kernel as the parameter. Moreover, the new pattern vectors should be constrained by the region covered by the current RBF kernel, and the initial filling of the RBF kernel can be accomplished in this way. Second, we need to measure the density of the region of the original pattern vectors in each RBF kernel. In the dense region of the sample space, the number of generated pattern instances is relatively large; conversely, in the sparse region of the sample space, the number of generated pattern instances is relatively small. When the generated instances are in the mixed region covered by different pattern classes, the probability of preserving the sample is further reduced. In this way, by combining the density and location information of the region, the optimal selection of the generated pattern instances can be completed without changing the probability density distribution of the original sample space.According to the above methods, we take the idea of KHLD as the prototype and approximate the idea of KHLD by filling and screening the pattern vector of each kernel. On the other hand, the KHLD of each RBF kernel is converted to learning and division of more pattern vectors, which can improve the sparse sample spatial distribution caused by a sample size that is too small or sample space dimension that is too high, and the classification accuracy of the classifier can be enhanced. Note that, due to the inhomogeneity of the sample distribution inside the kernel, the approximation of the idea of KHLD by filling and screening the pattern vector of each kernel can be considered a soft partition; that is, the final classified surface can pass through the kernel to improve the overlap of different pattern subclasses effectively. Thus, it is more conducive to the adjustment of actual classification surface parameters.In summary, the main contributions of this work are as follows:(1)
The idea of KHLD is introduced into the neural network classifier, and its characteristics are analyzed(2)
The internal sample generation and optimization screening mechanism of the RBF kernel is designed to achieve the approximation of KHLD(3)
The performance of KHLD is combined with existing classification algorithms and compared with these algorithms in two artificial datasets and several benchmark datasets, and the experimental results show the superiority of the proposed method


## 2. Methods

### 2.1. The Establishment of KHLD

Considering that the method of KHLD is based on the RBF kernel of HSARBF-ELM, here, we give the optimization of RBF kernels in HSARBF-ELM, which is ready for the optimization method of kernel holistic learning and division.For the input samplex, when it passes through the RBF kernel function, its output can be expressed as(1)φkx=exp−12σk2x−μk2,where μkandσk are the center and width of k-th RBF kernels.In HSARBF-ELM, by combining the methods of density clustering with a potential function and center-oriented unidirectional repulsive force, the numbers of RBF kernels and parameters can be effectively generated. The main methods are as follows.Given a training setS, where S=∪i=1hSi, Si is the i-th pattern category set, Si=x1i,…,xLii, h is the number of pattern categories, and Li is the number of samples in the i-th pattern analogy, for each pattern category set Si:(1)
Compute the potential value ofxvi according to(2)ρxvi=∑u=1,u≠vNi11+T⋅d2xui,xvi,v=1,2,…,Ni,whereT is the distance weighting factor and dxui,xvi is the distance measure between xui and xvi.(2)
Determine the sample with the maximum potential as the center of the hidden nodes of the generated RBF, and set the sample with the maximum potential to bexpi; the corresponding expression is as follows:(3)ρxpi=maxρx1i,ρx2i,…,ρxNii,μk=xpi.(3)
Adjust the center(4)μk′=μk+1M∑q=1MjFxqjs.t.Mi′≥Mi,Mj′≤Mj,where(5)Fxqj=exp−α⋅dxqj,μk⋅xqj−μkxqj−μk,whereFxqj denotes the heterogeneous repulsive force from xqj to μk, xqj is a heterogeneous sample covered by the current hidden nodes of the RBF:xqj∉Si and xqj−μm<λ⋅σ, where λ is the width covering factor, α is the repulsive force control factor, and M is the iteration step. Mi and Mj denote the number of samples covered by the current hidden nodes of the RBF before updating. Mi′andMj′ denote the number of samples covered by the current RBF hidden nodes after updating.(4)
The width is adjusted as follows:(6)σk=maxmindμk′,xqjβ,σmin,ifMj′>0,σ,ifMj′=0,whereβ is the width constraint factor, σmin is the constrained minimum width parameter, and σ is the initial width. This adjustment ensures the relative diversity of each generated RBF hidden node, which can achieve a balance between the coverage effect and the generalization performance.(5)
Counteract each sample potential of the region covered by the current RBF hidden node and find the sample with the maximum potential to generate the next RBF hidden node(7)ρ′xni=ρxni−ρxpi⋅exp−12σk2xni−xpi2,n=1,2,…,Ni,whereρ′xni is the updated potential value of xni.(6)
Set the iteration termination condition as follows:Ifmaxρ′x1i,ρ′x2i,…,ρ′xNii>δGo to Steps 2-4.ElseThe process of learning the current pattern category is complete. Go to learn other pattern categories.EndIfAccording to the above steps, the number of RBF hidden nodes and the center and width, which can be denoted asK,μk,σkk=1K, can be generated optimally. For HSRBF-ELM, once the optimized RBF hidden nodes are generated, the output gx=g1x,g2x,…,gjx,…,gKx can be the input of the subsequent ELM network. The update of the ELM network weights is based on the existing ELM [24] learning algorithm.


### 2.2. The Method of KHLD

#### 2.2.1. Main Idea

To explain the characteristics and advantages of the method based on KHLD, Figure1 gives a diagram of the direct classification and comparison of KHLD and direct pattern vector classification. The method of KHLD is transformed for training and dividing more pattern vectors.Figure 1
A schematic comparison between the kernel holistic partition and the direct pattern vector classification. (a) Directly partitioning the original sample set. (b) Density clustering of the original sample set and establishing the corresponding RBF kernels to complete the coverage of the original sample space. (c) Filling each subkernel pattern class to establish a new pattern vector to partition the whole kernel. (d) Dividing the original sample and the new filled sample into new sample sets to obtain a new classification surface. (e) Comparing the modified classification surface with the original one.
(a)(b)(c)(d)(e)To realize KHLD, it is necessary to establish a suitable RBF kernel to complete the effective coverage of the different regions in the original sample space. Then, to ensure the validity of the generated samples, the newly generated samples in the kernel should be approximately consistent with the original pattern vector distribution, and the number of newly generated samples should be proportional to the distribution density of the original sample region. In addition, when the kernels of different pattern categories overlap, it is necessary to further screen the generated pattern vector in overlapping regions. To this end, the following steps need to be completed:Step 1: the optimal coverage of the original sample space is completed by the potential function density clustering and center-oriented heterogeneous sample repulsive force; the appropriate RBF kernel parameters, including the number, center, and width of the RBF kernels, can be determined adaptively according to the distribution of the sample spaceStep 2: with the center and the width of each RBF kernel as constraints, a probability distribution similar to the original sample is set up to generate a new pattern vector in the effective region covered by each RBF kernelStep 3: the newly generated pattern vector is judged to determine whether it is retained or not, and finally, a new pattern vector subset is formedStep 4: w new set of samples is formed by combining the original sample with all the screened pattern vectors that are eventually retained to train the weights of the output classifierThe difficulty of realizing the above steps lies in Step 3, that is, to establish the appropriate standard to measure the relationship between the newly generated pattern vector and the original sample density and to determine whether the kernels with different pattern categories overlap each other to complete the optimization screening of the newly generated pattern vector.


#### 2.2.2. The Implementation

In this section, we first give the definitions of KHLD, overlapping region samples, and nonoverlapping region samples to prepare for the description and implementation of subsequent algorithms.(1) Definition.KHLD. Training and partitioning labeled RBF kernels after covering the original sample spaceOverlapping Region Samples. The samples in the overlapped region are covered by different pattern categories of RBF kernelsNonoverlapping Region Samples. Samples outside each overlapped regionAccording to the definition, Figure2 gives the schematic diagrams of the overlapping region samples and the nonoverlapping region samples, which represent the valid regions covered by two different RBF kernels. In Figure 2(b), C=A∩B, where C is the overlapping region, sample 1 and sample 2 are the overlapping region samples, and the other samples are nonoverlapping region samples.Figure 2
The schematic diagrams of distribution form of different kernels. (a) Nonoverlapping region samples. (b) Overlapping region samples.
(a)(b)To realize the classification method based on the kernel holistic division and the selection of generated samples, it is necessary to establish each RBF kernel as the research object and randomly generate the pattern category samples within each kernel to further optimize the screening process. To this end, two factors need to be considered:(1)
To facilitate the optimization of subsequent generated pattern samples, the probability distribution of the initial generated pattern samples should be approximately the same as that of the original sample.(2)
In the process of sample screening, the probability of the generated sample being retained should be proportional to the density of the original sample region. It is also necessary to consider whether the sample is an overlapping region sample and, if so, further reduce the probability that the sample is retained.For case (1), since the establishment of each RBF kernel parameter is based on the potential function density clustering, overall, the probability density of the region near the center of the original sample is relatively large, and the probability density of the region near the boundary of the original sample is relatively small, it can be considered that the probability density of pattern vectors in these kernels approximately obeys a multivariate Gaussian distribution with the current RBF kernel as the parameter, and it can be taken as the new pattern vector probability distribution model.For case (2), the key is to establish an appropriate measure to determine the density of the region where each generated sample is located and determine whether the generated sample is retained. If the generated sample is retained, it is necessary to determine whether the generated samples are in the overlapping region and further complete secondary optimization.According to the above description, given a datasetS=xi,yii=1L, where L is the number of training samples, yi is the category labelsxi∈Rl, and yi∈Rh, let Si be the training sample set of the i-th pattern category, Si=x1i,x2i,…,xLii; here, S=∪i=1hSi,Si∩Sj=∅, ∀i≠j. For each training sample category, the number and parameters of the RBF kernels are optimized by the potential function density and the repulsive force between heterogeneous samples, expressed as μki,σki,tkik=1Ki, where μki,σki are the center and width of the kth RBF kernel, respectively, tki is the ithpattern category label of the RBF kernel, Ki is the number of RBF kernels generated under each pattern category, and K=∑i=1hKi is the number of RBF kernels.When all the RBF kernels are built, the effective coverage of the different regions of the original sample space is completed. To achieve sample filling for each RBF kernel, it is necessary to establish a suitable sample probability distribution model to generate new pattern vectors. For the currentkth RBF kernel, the probability distribution fz for generating arbitrary pattern vectors zobeys the Gaussian distribution with μki being the mean and Σki=diagσki,σki,…,σki being the variance matrix; that is, z∼Nμki,Σki. Moreover, the newly generated pattern vectors should be in the effective region covered by the RBF kernel, which is given by(8)fz=12πσkiexp−z−μki22σk2s.t.z−μki≤σki.According to the above method, for thekth RBF kernel in the ith pattern category, let Wki be the generated initial vector set in the kernel; here, Wki=z1i,z2i,…,zNki, Nkis the number of generated samples in the k-th kernel. After the initial pattern vectors are generated, they need to be optimized and screened. During the screening process, in the dense region of the sample space, the number of generated pattern instances should be relatively large; conversely, in the sparse region of the sample space, the number of generated pattern instances should be relatively small. In this way, the probability distribution of the sample space can be combined with the density of the region where the pattern vector is generated, and the validity of the resulting pattern vector can be enhanced.LetCki be the initial sample set of the kthRBF kernel in the current ithpattern category and Pk be the number of Cki. For each initial pattern vector x, when x∈Si and x−μki≤σki, then x∈Cki. Thus, Cki=x1i,x2i,…,xPki. For each generated pattern vector zmi1≤m≤Nk in Wki, the probability density of each new pattern vector can be estimated, which is given as(9)p^zmi=1Pk∑j=1Pk12πl/2θklexp−zmi−xji22θk2,where θk is the width of the corresponding Parzen window in the k-th RBF kernel.To achieve this metric while preserving the randomness of sample generation, we consider generating a uniformly distributed random numberr between 0 and 1, which is used to for comparison with the probability density of each newly generated pattern vector. If r≤p^zmi, zmi is retained; otherwise, zmi is eliminated. Therefore, in the region where the original samples are relatively densely distributed, the probability that the newly generated samples will be retained is relatively high.Due to the complexity of different sample sets, heterogeneous samples are often mixed into the generated RBF kernel. Thus, it is necessary to further improve the sample screening in the overlapping region. When the generated sample is in the overlapping region, two factors need to be considered:(1)
The probability of the sample being retained should be reduced.(2)
It is necessary to consider the sample spatial distribution density under the current pattern category and other pattern categories at the same time. According to the principle of inhibiting the probability density of heterogeneous samples, when the spatial distribution density of the sample in the current pattern category is higher than that in other pattern categories, the probability of the sample being retained is relatively large.Combined with the above two factors, for the samplezmi generated in the kth RBF kernel, we can get zmi−μki≤σki. Moreover, when zmi−μnj≤σnj is satisfied, zmi can be considered the sample in the overlapping region between the k-th and the n-th RBF kernel.When the samples in the overlapping region are determined, it is necessary to further screen the samples. LetCnj be the initial sample set contained in the j-th pattern category by the kernel μnj,σnj, and set the number of samples in Cnj as Pn. For an arbitrary pattern vector x, when x∈Sj and x−μnj≤σnj are satisfied, x∈Cnj. Thus, Cnj=x1j,x2j,…,xPnj. For the sample zmi in the overlapping region intersecting the k-th and the n-th RBF kernel, the probability density estimations of the heterogeneous sample regions are expressed as(10)q^zmi=1Pn∑n=1Pn12πl/2θklexp−zmi−xnj22θk2.According to the above method, for a randomly generated numberr between 0 and 1, when p^zmi≥λr and p^zmi≥γq^zmi, the sample zmi in the overlapping region is retained; otherwise, zmi is removed. Here, λ>1 and γ≥1.Combined with the above description, Algorithm1 gives the concrete implementation of the classification method based on kernel holistic learning and kernel interior sample generation.Algorithm 1:Kernel holistic learning and kernel interior sample generation.
Initialization;fori=1:h % h is the number of pattern categoriesfork=1:KiCount the number of initial samplesPk belonging to the ith pattern category covered by each RBF hidden node;Use (8) to generate a sample set Wki=z1i,z2i,…,zNki and count the number of generated samples Nk;form=1:Nk % Screening of generated samples according to the densityUse (9) to estimate the probability density p^zmi belonging to the current ithpattern category;r=rand1;ifp^zmi<rCki=Cki−zmi;end ifend forupdateNk;end forform=1:Nk %further screening of the overlapping region samplesforn=1:K−Kiifzmi−μnj<σnjUse (10) to estimate the probability density q^zmi belonging to the jth pattern category;ifp^zmi≤q^zmi‖p^zmi≤γrCki=Cki−zmi;end ifend ifend forend forend for


#### 2.2.3. The Computational Complexity Analysis of KHLD

In this study, a method of the potential density clustering and the center-oriented heterogeneous sample repulsive force is used to generate optimized kernel parameters. Then, a method of optimized sample filling and screening can realize the effective approximation of KHLD. Assume that the number of samples in the initial training setS is L, and the initial training set contains two pattern categories; the number of samples are L1 and L2, respectively. Here, L1+L2=L. The computational complexity of the proposed method is analyzed as follows:(1)
The optimal kernel parameters are generated by the combination of potential density clustering and heterogeneous sample repulsive force. In the process of quantifying the sample potential value by potential function density clustering, the label information of each category of samples is considered. The calculation of the sample potential value needs to traverse all other samples in the current pattern category. Then, Gaussian kernels with different parameters are needed to cover the sample subspace to update the sample potential. The computational complexity isOL1−12+L2−12. Set the number of kernels as K; in the process of optimizing the kernel parameters, the distance between all samples and the center should be considered; the computational complexity is OLK. After merging, the computational complexity of this part is OL2−2L1L2−2L+LK.(2)
The process of sample generation and screening will also take a certain amount of time. Let the number of samples generated in all kernels beP, where the number of samples generated in the k-th kernel is Pk; thus, P=∑k=1KPk. In the process of calculating the density measurement of the generated sample, the distance between the generated sample and the center of the current kernel should be considered; here, the computational complexity of the generated sample in the k-th kernel is OPk. The computational complexity of all kernel generated samples is combined, which can be expressed as O∑k=1KPk; here, O∑k=1KPk=OP. Then, in the process of sample screening, we need to further consider whether the generated samples in the current kernel are overlapping region samples, which requires us to compare the distance between these samples and all other centers, and the computational complexity is OK−1Pk. The computational complexity of all kernel screening samples is combined, which can be expressed as O∑k=1KK−1⋅Pk. Thus, the computational complexity of sample generation and screening in all established kernels is OP+O∑k=1KK−1⋅Pk, which can be simplified as OKP.Combined with Steps 1 and 2, the computational complexity of the proposed KHLD isOL2−2L1L2−2L+LK+KP. Then, the generated training samples and the original training samples are combined to complete the training of the existing algorithms.


## 2.1. The Establishment of KHLD

Considering that the method of KHLD is based on the RBF kernel of HSARBF-ELM, here, we give the optimization of RBF kernels in HSARBF-ELM, which is ready for the optimization method of kernel holistic learning and division.For the input samplex, when it passes through the RBF kernel function, its output can be expressed as(1)φkx=exp−12σk2x−μk2,where μkandσk are the center and width of k-th RBF kernels.In HSARBF-ELM, by combining the methods of density clustering with a potential function and center-oriented unidirectional repulsive force, the numbers of RBF kernels and parameters can be effectively generated. The main methods are as follows.Given a training setS, where S=∪i=1hSi, Si is the i-th pattern category set, Si=x1i,…,xLii, h is the number of pattern categories, and Li is the number of samples in the i-th pattern analogy, for each pattern category set Si:(1)
Compute the potential value ofxvi according to(2)ρxvi=∑u=1,u≠vNi11+T⋅d2xui,xvi,v=1,2,…,Ni,whereT is the distance weighting factor and dxui,xvi is the distance measure between xui and xvi.(2)
Determine the sample with the maximum potential as the center of the hidden nodes of the generated RBF, and set the sample with the maximum potential to bexpi; the corresponding expression is as follows:(3)ρxpi=maxρx1i,ρx2i,…,ρxNii,μk=xpi.(3)
Adjust the center(4)μk′=μk+1M∑q=1MjFxqjs.t.Mi′≥Mi,Mj′≤Mj,where(5)Fxqj=exp−α⋅dxqj,μk⋅xqj−μkxqj−μk,whereFxqj denotes the heterogeneous repulsive force from xqj to μk, xqj is a heterogeneous sample covered by the current hidden nodes of the RBF:xqj∉Si and xqj−μm<λ⋅σ, where λ is the width covering factor, α is the repulsive force control factor, and M is the iteration step. Mi and Mj denote the number of samples covered by the current hidden nodes of the RBF before updating. Mi′andMj′ denote the number of samples covered by the current RBF hidden nodes after updating.(4)
The width is adjusted as follows:(6)σk=maxmindμk′,xqjβ,σmin,ifMj′>0,σ,ifMj′=0,whereβ is the width constraint factor, σmin is the constrained minimum width parameter, and σ is the initial width. This adjustment ensures the relative diversity of each generated RBF hidden node, which can achieve a balance between the coverage effect and the generalization performance.(5)
Counteract each sample potential of the region covered by the current RBF hidden node and find the sample with the maximum potential to generate the next RBF hidden node(7)ρ′xni=ρxni−ρxpi⋅exp−12σk2xni−xpi2,n=1,2,…,Ni,whereρ′xni is the updated potential value of xni.(6)
Set the iteration termination condition as follows:Ifmaxρ′x1i,ρ′x2i,…,ρ′xNii>δGo to Steps 2-4.ElseThe process of learning the current pattern category is complete. Go to learn other pattern categories.EndIfAccording to the above steps, the number of RBF hidden nodes and the center and width, which can be denoted asK,μk,σkk=1K, can be generated optimally. For HSRBF-ELM, once the optimized RBF hidden nodes are generated, the output gx=g1x,g2x,…,gjx,…,gKx can be the input of the subsequent ELM network. The update of the ELM network weights is based on the existing ELM [24] learning algorithm.


## 2.2. The Method of KHLD

### 2.2.1. Main Idea

To explain the characteristics and advantages of the method based on KHLD, Figure1 gives a diagram of the direct classification and comparison of KHLD and direct pattern vector classification. The method of KHLD is transformed for training and dividing more pattern vectors.Figure 1
A schematic comparison between the kernel holistic partition and the direct pattern vector classification. (a) Directly partitioning the original sample set. (b) Density clustering of the original sample set and establishing the corresponding RBF kernels to complete the coverage of the original sample space. (c) Filling each subkernel pattern class to establish a new pattern vector to partition the whole kernel. (d) Dividing the original sample and the new filled sample into new sample sets to obtain a new classification surface. (e) Comparing the modified classification surface with the original one.
(a)(b)(c)(d)(e)To realize KHLD, it is necessary to establish a suitable RBF kernel to complete the effective coverage of the different regions in the original sample space. Then, to ensure the validity of the generated samples, the newly generated samples in the kernel should be approximately consistent with the original pattern vector distribution, and the number of newly generated samples should be proportional to the distribution density of the original sample region. In addition, when the kernels of different pattern categories overlap, it is necessary to further screen the generated pattern vector in overlapping regions. To this end, the following steps need to be completed:Step 1: the optimal coverage of the original sample space is completed by the potential function density clustering and center-oriented heterogeneous sample repulsive force; the appropriate RBF kernel parameters, including the number, center, and width of the RBF kernels, can be determined adaptively according to the distribution of the sample spaceStep 2: with the center and the width of each RBF kernel as constraints, a probability distribution similar to the original sample is set up to generate a new pattern vector in the effective region covered by each RBF kernelStep 3: the newly generated pattern vector is judged to determine whether it is retained or not, and finally, a new pattern vector subset is formedStep 4: w new set of samples is formed by combining the original sample with all the screened pattern vectors that are eventually retained to train the weights of the output classifierThe difficulty of realizing the above steps lies in Step 3, that is, to establish the appropriate standard to measure the relationship between the newly generated pattern vector and the original sample density and to determine whether the kernels with different pattern categories overlap each other to complete the optimization screening of the newly generated pattern vector.


### 2.2.2. The Implementation

In this section, we first give the definitions of KHLD, overlapping region samples, and nonoverlapping region samples to prepare for the description and implementation of subsequent algorithms.(1) Definition.KHLD. Training and partitioning labeled RBF kernels after covering the original sample spaceOverlapping Region Samples. The samples in the overlapped region are covered by different pattern categories of RBF kernelsNonoverlapping Region Samples. Samples outside each overlapped regionAccording to the definition, Figure2 gives the schematic diagrams of the overlapping region samples and the nonoverlapping region samples, which represent the valid regions covered by two different RBF kernels. In Figure 2(b), C=A∩B, where C is the overlapping region, sample 1 and sample 2 are the overlapping region samples, and the other samples are nonoverlapping region samples.Figure 2
The schematic diagrams of distribution form of different kernels. (a) Nonoverlapping region samples. (b) Overlapping region samples.
(a)(b)To realize the classification method based on the kernel holistic division and the selection of generated samples, it is necessary to establish each RBF kernel as the research object and randomly generate the pattern category samples within each kernel to further optimize the screening process. To this end, two factors need to be considered:(1)
To facilitate the optimization of subsequent generated pattern samples, the probability distribution of the initial generated pattern samples should be approximately the same as that of the original sample.(2)
In the process of sample screening, the probability of the generated sample being retained should be proportional to the density of the original sample region. It is also necessary to consider whether the sample is an overlapping region sample and, if so, further reduce the probability that the sample is retained.For case (1), since the establishment of each RBF kernel parameter is based on the potential function density clustering, overall, the probability density of the region near the center of the original sample is relatively large, and the probability density of the region near the boundary of the original sample is relatively small, it can be considered that the probability density of pattern vectors in these kernels approximately obeys a multivariate Gaussian distribution with the current RBF kernel as the parameter, and it can be taken as the new pattern vector probability distribution model.For case (2), the key is to establish an appropriate measure to determine the density of the region where each generated sample is located and determine whether the generated sample is retained. If the generated sample is retained, it is necessary to determine whether the generated samples are in the overlapping region and further complete secondary optimization.According to the above description, given a datasetS=xi,yii=1L, where L is the number of training samples, yi is the category labelsxi∈Rl, and yi∈Rh, let Si be the training sample set of the i-th pattern category, Si=x1i,x2i,…,xLii; here, S=∪i=1hSi,Si∩Sj=∅, ∀i≠j. For each training sample category, the number and parameters of the RBF kernels are optimized by the potential function density and the repulsive force between heterogeneous samples, expressed as μki,σki,tkik=1Ki, where μki,σki are the center and width of the kth RBF kernel, respectively, tki is the ithpattern category label of the RBF kernel, Ki is the number of RBF kernels generated under each pattern category, and K=∑i=1hKi is the number of RBF kernels.When all the RBF kernels are built, the effective coverage of the different regions of the original sample space is completed. To achieve sample filling for each RBF kernel, it is necessary to establish a suitable sample probability distribution model to generate new pattern vectors. For the currentkth RBF kernel, the probability distribution fz for generating arbitrary pattern vectors zobeys the Gaussian distribution with μki being the mean and Σki=diagσki,σki,…,σki being the variance matrix; that is, z∼Nμki,Σki. Moreover, the newly generated pattern vectors should be in the effective region covered by the RBF kernel, which is given by(8)fz=12πσkiexp−z−μki22σk2s.t.z−μki≤σki.According to the above method, for thekth RBF kernel in the ith pattern category, let Wki be the generated initial vector set in the kernel; here, Wki=z1i,z2i,…,zNki, Nkis the number of generated samples in the k-th kernel. After the initial pattern vectors are generated, they need to be optimized and screened. During the screening process, in the dense region of the sample space, the number of generated pattern instances should be relatively large; conversely, in the sparse region of the sample space, the number of generated pattern instances should be relatively small. In this way, the probability distribution of the sample space can be combined with the density of the region where the pattern vector is generated, and the validity of the resulting pattern vector can be enhanced.LetCki be the initial sample set of the kthRBF kernel in the current ithpattern category and Pk be the number of Cki. For each initial pattern vector x, when x∈Si and x−μki≤σki, then x∈Cki. Thus, Cki=x1i,x2i,…,xPki. For each generated pattern vector zmi1≤m≤Nk in Wki, the probability density of each new pattern vector can be estimated, which is given as(9)p^zmi=1Pk∑j=1Pk12πl/2θklexp−zmi−xji22θk2,where θk is the width of the corresponding Parzen window in the k-th RBF kernel.To achieve this metric while preserving the randomness of sample generation, we consider generating a uniformly distributed random numberr between 0 and 1, which is used to for comparison with the probability density of each newly generated pattern vector. If r≤p^zmi, zmi is retained; otherwise, zmi is eliminated. Therefore, in the region where the original samples are relatively densely distributed, the probability that the newly generated samples will be retained is relatively high.Due to the complexity of different sample sets, heterogeneous samples are often mixed into the generated RBF kernel. Thus, it is necessary to further improve the sample screening in the overlapping region. When the generated sample is in the overlapping region, two factors need to be considered:(1)
The probability of the sample being retained should be reduced.(2)
It is necessary to consider the sample spatial distribution density under the current pattern category and other pattern categories at the same time. According to the principle of inhibiting the probability density of heterogeneous samples, when the spatial distribution density of the sample in the current pattern category is higher than that in other pattern categories, the probability of the sample being retained is relatively large.Combined with the above two factors, for the samplezmi generated in the kth RBF kernel, we can get zmi−μki≤σki. Moreover, when zmi−μnj≤σnj is satisfied, zmi can be considered the sample in the overlapping region between the k-th and the n-th RBF kernel.When the samples in the overlapping region are determined, it is necessary to further screen the samples. LetCnj be the initial sample set contained in the j-th pattern category by the kernel μnj,σnj, and set the number of samples in Cnj as Pn. For an arbitrary pattern vector x, when x∈Sj and x−μnj≤σnj are satisfied, x∈Cnj. Thus, Cnj=x1j,x2j,…,xPnj. For the sample zmi in the overlapping region intersecting the k-th and the n-th RBF kernel, the probability density estimations of the heterogeneous sample regions are expressed as(10)q^zmi=1Pn∑n=1Pn12πl/2θklexp−zmi−xnj22θk2.According to the above method, for a randomly generated numberr between 0 and 1, when p^zmi≥λr and p^zmi≥γq^zmi, the sample zmi in the overlapping region is retained; otherwise, zmi is removed. Here, λ>1 and γ≥1.Combined with the above description, Algorithm1 gives the concrete implementation of the classification method based on kernel holistic learning and kernel interior sample generation.Algorithm 1:Kernel holistic learning and kernel interior sample generation.
Initialization;fori=1:h % h is the number of pattern categoriesfork=1:KiCount the number of initial samplesPk belonging to the ith pattern category covered by each RBF hidden node;Use (8) to generate a sample set Wki=z1i,z2i,…,zNki and count the number of generated samples Nk;form=1:Nk % Screening of generated samples according to the densityUse (9) to estimate the probability density p^zmi belonging to the current ithpattern category;r=rand1;ifp^zmi<rCki=Cki−zmi;end ifend forupdateNk;end forform=1:Nk %further screening of the overlapping region samplesforn=1:K−Kiifzmi−μnj<σnjUse (10) to estimate the probability density q^zmi belonging to the jth pattern category;ifp^zmi≤q^zmi‖p^zmi≤γrCki=Cki−zmi;end ifend ifend forend forend for


### 2.2.3. The Computational Complexity Analysis of KHLD

In this study, a method of the potential density clustering and the center-oriented heterogeneous sample repulsive force is used to generate optimized kernel parameters. Then, a method of optimized sample filling and screening can realize the effective approximation of KHLD. Assume that the number of samples in the initial training setS is L, and the initial training set contains two pattern categories; the number of samples are L1 and L2, respectively. Here, L1+L2=L. The computational complexity of the proposed method is analyzed as follows:(1)
The optimal kernel parameters are generated by the combination of potential density clustering and heterogeneous sample repulsive force. In the process of quantifying the sample potential value by potential function density clustering, the label information of each category of samples is considered. The calculation of the sample potential value needs to traverse all other samples in the current pattern category. Then, Gaussian kernels with different parameters are needed to cover the sample subspace to update the sample potential. The computational complexity isOL1−12+L2−12. Set the number of kernels as K; in the process of optimizing the kernel parameters, the distance between all samples and the center should be considered; the computational complexity is OLK. After merging, the computational complexity of this part is OL2−2L1L2−2L+LK.(2)
The process of sample generation and screening will also take a certain amount of time. Let the number of samples generated in all kernels beP, where the number of samples generated in the k-th kernel is Pk; thus, P=∑k=1KPk. In the process of calculating the density measurement of the generated sample, the distance between the generated sample and the center of the current kernel should be considered; here, the computational complexity of the generated sample in the k-th kernel is OPk. The computational complexity of all kernel generated samples is combined, which can be expressed as O∑k=1KPk; here, O∑k=1KPk=OP. Then, in the process of sample screening, we need to further consider whether the generated samples in the current kernel are overlapping region samples, which requires us to compare the distance between these samples and all other centers, and the computational complexity is OK−1Pk. The computational complexity of all kernel screening samples is combined, which can be expressed as O∑k=1KK−1⋅Pk. Thus, the computational complexity of sample generation and screening in all established kernels is OP+O∑k=1KK−1⋅Pk, which can be simplified as OKP.Combined with Steps 1 and 2, the computational complexity of the proposed KHLD isOL2−2L1L2−2L+LK+KP. Then, the generated training samples and the original training samples are combined to complete the training of the existing algorithms.


## 2.2.1. Main Idea

To explain the characteristics and advantages of the method based on KHLD, Figure1 gives a diagram of the direct classification and comparison of KHLD and direct pattern vector classification. The method of KHLD is transformed for training and dividing more pattern vectors.Figure 1
A schematic comparison between the kernel holistic partition and the direct pattern vector classification. (a) Directly partitioning the original sample set. (b) Density clustering of the original sample set and establishing the corresponding RBF kernels to complete the coverage of the original sample space. (c) Filling each subkernel pattern class to establish a new pattern vector to partition the whole kernel. (d) Dividing the original sample and the new filled sample into new sample sets to obtain a new classification surface. (e) Comparing the modified classification surface with the original one.
(a)(b)(c)(d)(e)To realize KHLD, it is necessary to establish a suitable RBF kernel to complete the effective coverage of the different regions in the original sample space. Then, to ensure the validity of the generated samples, the newly generated samples in the kernel should be approximately consistent with the original pattern vector distribution, and the number of newly generated samples should be proportional to the distribution density of the original sample region. In addition, when the kernels of different pattern categories overlap, it is necessary to further screen the generated pattern vector in overlapping regions. To this end, the following steps need to be completed:Step 1: the optimal coverage of the original sample space is completed by the potential function density clustering and center-oriented heterogeneous sample repulsive force; the appropriate RBF kernel parameters, including the number, center, and width of the RBF kernels, can be determined adaptively according to the distribution of the sample spaceStep 2: with the center and the width of each RBF kernel as constraints, a probability distribution similar to the original sample is set up to generate a new pattern vector in the effective region covered by each RBF kernelStep 3: the newly generated pattern vector is judged to determine whether it is retained or not, and finally, a new pattern vector subset is formedStep 4: w new set of samples is formed by combining the original sample with all the screened pattern vectors that are eventually retained to train the weights of the output classifierThe difficulty of realizing the above steps lies in Step 3, that is, to establish the appropriate standard to measure the relationship between the newly generated pattern vector and the original sample density and to determine whether the kernels with different pattern categories overlap each other to complete the optimization screening of the newly generated pattern vector.


## 2.2.2. The Implementation

In this section, we first give the definitions of KHLD, overlapping region samples, and nonoverlapping region samples to prepare for the description and implementation of subsequent algorithms.(1) Definition.KHLD. Training and partitioning labeled RBF kernels after covering the original sample spaceOverlapping Region Samples. The samples in the overlapped region are covered by different pattern categories of RBF kernelsNonoverlapping Region Samples. Samples outside each overlapped regionAccording to the definition, Figure2 gives the schematic diagrams of the overlapping region samples and the nonoverlapping region samples, which represent the valid regions covered by two different RBF kernels. In Figure 2(b), C=A∩B, where C is the overlapping region, sample 1 and sample 2 are the overlapping region samples, and the other samples are nonoverlapping region samples.Figure 2
The schematic diagrams of distribution form of different kernels. (a) Nonoverlapping region samples. (b) Overlapping region samples.
(a)(b)To realize the classification method based on the kernel holistic division and the selection of generated samples, it is necessary to establish each RBF kernel as the research object and randomly generate the pattern category samples within each kernel to further optimize the screening process. To this end, two factors need to be considered:(1)
To facilitate the optimization of subsequent generated pattern samples, the probability distribution of the initial generated pattern samples should be approximately the same as that of the original sample.(2)
In the process of sample screening, the probability of the generated sample being retained should be proportional to the density of the original sample region. It is also necessary to consider whether the sample is an overlapping region sample and, if so, further reduce the probability that the sample is retained.For case (1), since the establishment of each RBF kernel parameter is based on the potential function density clustering, overall, the probability density of the region near the center of the original sample is relatively large, and the probability density of the region near the boundary of the original sample is relatively small, it can be considered that the probability density of pattern vectors in these kernels approximately obeys a multivariate Gaussian distribution with the current RBF kernel as the parameter, and it can be taken as the new pattern vector probability distribution model.For case (2), the key is to establish an appropriate measure to determine the density of the region where each generated sample is located and determine whether the generated sample is retained. If the generated sample is retained, it is necessary to determine whether the generated samples are in the overlapping region and further complete secondary optimization.According to the above description, given a datasetS=xi,yii=1L, where L is the number of training samples, yi is the category labelsxi∈Rl, and yi∈Rh, let Si be the training sample set of the i-th pattern category, Si=x1i,x2i,…,xLii; here, S=∪i=1hSi,Si∩Sj=∅, ∀i≠j. For each training sample category, the number and parameters of the RBF kernels are optimized by the potential function density and the repulsive force between heterogeneous samples, expressed as μki,σki,tkik=1Ki, where μki,σki are the center and width of the kth RBF kernel, respectively, tki is the ithpattern category label of the RBF kernel, Ki is the number of RBF kernels generated under each pattern category, and K=∑i=1hKi is the number of RBF kernels.When all the RBF kernels are built, the effective coverage of the different regions of the original sample space is completed. To achieve sample filling for each RBF kernel, it is necessary to establish a suitable sample probability distribution model to generate new pattern vectors. For the currentkth RBF kernel, the probability distribution fz for generating arbitrary pattern vectors zobeys the Gaussian distribution with μki being the mean and Σki=diagσki,σki,…,σki being the variance matrix; that is, z∼Nμki,Σki. Moreover, the newly generated pattern vectors should be in the effective region covered by the RBF kernel, which is given by(8)fz=12πσkiexp−z−μki22σk2s.t.z−μki≤σki.According to the above method, for thekth RBF kernel in the ith pattern category, let Wki be the generated initial vector set in the kernel; here, Wki=z1i,z2i,…,zNki, Nkis the number of generated samples in the k-th kernel. After the initial pattern vectors are generated, they need to be optimized and screened. During the screening process, in the dense region of the sample space, the number of generated pattern instances should be relatively large; conversely, in the sparse region of the sample space, the number of generated pattern instances should be relatively small. In this way, the probability distribution of the sample space can be combined with the density of the region where the pattern vector is generated, and the validity of the resulting pattern vector can be enhanced.LetCki be the initial sample set of the kthRBF kernel in the current ithpattern category and Pk be the number of Cki. For each initial pattern vector x, when x∈Si and x−μki≤σki, then x∈Cki. Thus, Cki=x1i,x2i,…,xPki. For each generated pattern vector zmi1≤m≤Nk in Wki, the probability density of each new pattern vector can be estimated, which is given as(9)p^zmi=1Pk∑j=1Pk12πl/2θklexp−zmi−xji22θk2,where θk is the width of the corresponding Parzen window in the k-th RBF kernel.To achieve this metric while preserving the randomness of sample generation, we consider generating a uniformly distributed random numberr between 0 and 1, which is used to for comparison with the probability density of each newly generated pattern vector. If r≤p^zmi, zmi is retained; otherwise, zmi is eliminated. Therefore, in the region where the original samples are relatively densely distributed, the probability that the newly generated samples will be retained is relatively high.Due to the complexity of different sample sets, heterogeneous samples are often mixed into the generated RBF kernel. Thus, it is necessary to further improve the sample screening in the overlapping region. When the generated sample is in the overlapping region, two factors need to be considered:(1)
The probability of the sample being retained should be reduced.(2)
It is necessary to consider the sample spatial distribution density under the current pattern category and other pattern categories at the same time. According to the principle of inhibiting the probability density of heterogeneous samples, when the spatial distribution density of the sample in the current pattern category is higher than that in other pattern categories, the probability of the sample being retained is relatively large.Combined with the above two factors, for the samplezmi generated in the kth RBF kernel, we can get zmi−μki≤σki. Moreover, when zmi−μnj≤σnj is satisfied, zmi can be considered the sample in the overlapping region between the k-th and the n-th RBF kernel.When the samples in the overlapping region are determined, it is necessary to further screen the samples. LetCnj be the initial sample set contained in the j-th pattern category by the kernel μnj,σnj, and set the number of samples in Cnj as Pn. For an arbitrary pattern vector x, when x∈Sj and x−μnj≤σnj are satisfied, x∈Cnj. Thus, Cnj=x1j,x2j,…,xPnj. For the sample zmi in the overlapping region intersecting the k-th and the n-th RBF kernel, the probability density estimations of the heterogeneous sample regions are expressed as(10)q^zmi=1Pn∑n=1Pn12πl/2θklexp−zmi−xnj22θk2.According to the above method, for a randomly generated numberr between 0 and 1, when p^zmi≥λr and p^zmi≥γq^zmi, the sample zmi in the overlapping region is retained; otherwise, zmi is removed. Here, λ>1 and γ≥1.Combined with the above description, Algorithm1 gives the concrete implementation of the classification method based on kernel holistic learning and kernel interior sample generation.Algorithm 1:Kernel holistic learning and kernel interior sample generation.
Initialization;fori=1:h % h is the number of pattern categoriesfork=1:KiCount the number of initial samplesPk belonging to the ith pattern category covered by each RBF hidden node;Use (8) to generate a sample set Wki=z1i,z2i,…,zNki and count the number of generated samples Nk;form=1:Nk % Screening of generated samples according to the densityUse (9) to estimate the probability density p^zmi belonging to the current ithpattern category;r=rand1;ifp^zmi<rCki=Cki−zmi;end ifend forupdateNk;end forform=1:Nk %further screening of the overlapping region samplesforn=1:K−Kiifzmi−μnj<σnjUse (10) to estimate the probability density q^zmi belonging to the jth pattern category;ifp^zmi≤q^zmi‖p^zmi≤γrCki=Cki−zmi;end ifend ifend forend forend for


## 2.2.3. The Computational Complexity Analysis of KHLD

In this study, a method of the potential density clustering and the center-oriented heterogeneous sample repulsive force is used to generate optimized kernel parameters. Then, a method of optimized sample filling and screening can realize the effective approximation of KHLD. Assume that the number of samples in the initial training setS is L, and the initial training set contains two pattern categories; the number of samples are L1 and L2, respectively. Here, L1+L2=L. The computational complexity of the proposed method is analyzed as follows:(1)
The optimal kernel parameters are generated by the combination of potential density clustering and heterogeneous sample repulsive force. In the process of quantifying the sample potential value by potential function density clustering, the label information of each category of samples is considered. The calculation of the sample potential value needs to traverse all other samples in the current pattern category. Then, Gaussian kernels with different parameters are needed to cover the sample subspace to update the sample potential. The computational complexity isOL1−12+L2−12. Set the number of kernels as K; in the process of optimizing the kernel parameters, the distance between all samples and the center should be considered; the computational complexity is OLK. After merging, the computational complexity of this part is OL2−2L1L2−2L+LK.(2)
The process of sample generation and screening will also take a certain amount of time. Let the number of samples generated in all kernels beP, where the number of samples generated in the k-th kernel is Pk; thus, P=∑k=1KPk. In the process of calculating the density measurement of the generated sample, the distance between the generated sample and the center of the current kernel should be considered; here, the computational complexity of the generated sample in the k-th kernel is OPk. The computational complexity of all kernel generated samples is combined, which can be expressed as O∑k=1KPk; here, O∑k=1KPk=OP. Then, in the process of sample screening, we need to further consider whether the generated samples in the current kernel are overlapping region samples, which requires us to compare the distance between these samples and all other centers, and the computational complexity is OK−1Pk. The computational complexity of all kernel screening samples is combined, which can be expressed as O∑k=1KK−1⋅Pk. Thus, the computational complexity of sample generation and screening in all established kernels is OP+O∑k=1KK−1⋅Pk, which can be simplified as OKP.Combined with Steps 1 and 2, the computational complexity of the proposed KHLD isOL2−2L1L2−2L+LK+KP. Then, the generated training samples and the original training samples are combined to complete the training of the existing algorithms.


## 3. Results and Discussion

In this section, the performance of KHLD is evaluated with two artificial datasets: Double Moon (DM) [25] and Concrete Circle (CC); 8 UCI benchmark datasets [26]: Blood, Climate, Heart Disease (HD), Sonar, SPECT Heart (SH), Image Segmentation (IS), Forest, and Wilt; and 1 LIBSVM benchmark dataset [27]. Figure 3 shows the graphical display of two artificial datasets. Except for the DM, CC, and IS datasets, all benchmark datasets are imbalanced datasets. In each dataset, the inputs to all the classifiers are scaled to appropriately [−1, 1]; the classification performance of each network is measured by the overall ηo and average ηa per-category classification accuracies [23]. Table 1 gives the description of the classification datasets.Figure 3
Artificial datasets classification problem. (a) Double Moon. (b) Concrete Circle.
(a)(b)Table 1
Descriptions of the classification datasets.
DatasetsNo. of featuresNo. of classesNo. of samplesTrainingValidationTestingDM2250–3502003000CC2250–3502003000Blood42374187187Climate182270135135HD1321517676Sonar6021045252SH2228047140IS19721010001100Forest274198100225Svmguide162308915002500Wilt4230001339500The performance of KHLD is combined with existing classification algorithms and compared with these algorithms, including SVM [27], ELM [24], HSARBF-ELM, a constrained optimization method based on BP neural network (CO-BP) [28], and an optimized RBF network based on fractional order gradient descent with momentum (FOGDM-RBF) [29]. For SVM, the simulations are implemented with LIBSVM [27]. All these simulations are conducted in MATLAB R2013b running on a PC with 3.2 GHz CPU and 4G RAM. Each algorithm is conducted in 20 trials.


### 3.1. Artificial Datasets: DM and CC

In this section, two artificial datasets are used to verify the graphical and intuitive characteristics of KHLD. In the phase of classification performance comparison, KHLD is combined with HSARBF-ELM and compared with HSARBF-ELM. Figures4(a)–4(d) give a comparison of the learning and classification effects based on the original training set and the KHLD under the DM dataset. It can be seen that the RBF kernel generated in HSARBF-ELM can effectively cover the sample space. The combination of KHLD and HSARBF-ELM can fill the training sample space and effectively improve the classification performance of the method of HSARBF-ELM.Figure 4
The learning and classification comparison of the HSARBF-ELM network classifier based on the original training set and kernel holistic learning and division in the DM dataset, where the number of original training sets is 100 and the initial kernel width is 0.1. (a) Learning the original training set to generate different RBF kernels. (b) Further learning and screening on the basis of each RBF kernel to generate new sample vectors. (c) Classification effect obtained by learning the parameters of the classifier using the original training set. (d) Classification effect obtained by merging the original sample set with the newly generated sample set and learning the classifier parameters.
(a)(b)(c)(d)Figures5 and 6 show the optimization effect of the kernels and the samples generated in each kernel after adjusting parameters σ and θk, respectively, which shows that the adjustment of parameters σ and θk has good adaptability to the sample space on DM dataset.Figure 5
The effect comparison of kernel holistic learning when the number of original training samples is 100 and the initial kernel width takes different parameters. (a)σ=0.1. (b) σ=0.3. (c) σ=0.4.
(a)(b)(c)Figure 6
Comparison of the kernel holistic learning effect under different Parzen window width parameters. (a)θk=σk/5. (b) θk=σk/10. (c) θk=σk/20.
(a)(b)(c)Figure7 compares the number of samples generated and the classification accuracy under different initial training sets, where θk=δk/10. It can be seen that, under the condition of small number of training samples, when the initial kernel width is too small, the established kernel cannot effectively cover the sample space, which leads to the decline of network generalization performance; when the kernel width is large and the number of training samples is sufficient, the performance of the proposed method will also show a certain degree of decline, which shows that the method of KHLD has certain restrictions on the number of training samples and the selection of kernel width parameter.Figure 7
Comparison of the number of training samples and the classification accuracy under different initial training sets in the DM dataset. (a) Number of original training samples, number of generated training samples. (b) Number of original training samples, overall testing accuracy (%).
(a)(b)Figure8 shows the learning and classification comparison of the HASRBF-ELM network classifier based on the original training set and KHLD under the CC dataset. It can be seen that when the generated kernels of different categories overlap each other seriously, the proposed method can still generate new samples in different kernels and improve the classification performance of the original HASRBF-ELM network classifier, which shows the effectiveness of KHLD method for complex classification problems.Figure 8
Sample generation of the original training set under the CC dataset and the comparison of the classification results on the test set. (a) Learning the original training set to generate the subkernel. (b) Further learning and screening to generate new sample vectors. (c) Classification results obtained by using the original training set to learn the classifier parameters. (d) Classifying the original training set with the newly generated training set and then learning the classifier parameters.
(a)(b)(c)(d)Figure9 shows the learning effect of the proposed method on the training set as the initial kernel width varies. By changing different kernel width parameters, the method of KHLD can optimize the selection of samples in each kernel. When the kernel width increases, the generated kernels may cover the heterogeneous samples, resulting in the increase of the overlapping of the samples of different pattern categories in the kernel.Figure 9
Comparison of the effect of sample generation when the number of original training samples is 100 and the initial kernel width is different. (a)σ=0.1. (b) σ=0.2. (c) σ=0.3.
(a)(b)(c)Figure10 further shows the number of generated training samples and the performance comparison of the classification accuracy under different initial training sets, where θk=δk/5. When the width parameters of the RBF kernels are in a certain range, the method of KHLD has a good classification effect. Similar to Figure 7(b), when the initial kernel width is too small, the testing accuracy of the proposed method is greatly reduced, which means that the failure of the initial RBF kernel may invalidate the method of kernel holistic learning and further deteriorate the final classification performance. Thus, it is necessary to avoid such a situation. This condition is also a restrictive condition for KHLD in this study.Figure 10
Comparison of the classification performance and the number of generated training samples on the CC dataset. (a) Number of original training samples, number of generated training samples. (b) Number of original training samples, overall testing accuracy (%).
(a)(b)Figures11 and 12 show that the combination of KHLD and HSARBF-ELM increases the training time. However, the proposed method improves the network classification performance of HSARBF-ELM, especially when the number of training samples is small. When the number of training samples is sufficient, the proposed method will reduce the performance of HSARBF-ELM to a certain extent, which shows that this method of KHLD is suitable for the situation with less training samples or sparse spatial distribution of samples.Figure 11
Performance comparison of the proposed method and HSARBF-ELM in the DM dataset. (a) Number of original training samples, training time. (b) Number of original training samples, overall testing accuracy (%).
(a)(b)Figure 12
Performance comparison of the proposed method and HSARBF-ELM in the CC dataset. (a) Number of original training samples, training time. (b) Number of original training samples, overall testing accuracy (%).
(a)(b)


### 3.2. UCI Benchmark Datasets

Tables2 and 3 give the comparisons of the classification performance of the proposed method and other learning algorithms under the benchmark sample datasets. It can be seen that, in high-dimensional small sample datasets, the combination of KHLD and other classification algorithms increases the training time. Although the testing results of different classification algorithms on different datasets are different, the combination of KHLD and these classification algorithms can improve the testing accuracy of these algorithms to varying degrees. As an auxiliary method, KHLD is an effective method when the spatial distribution of samples is sparse. The effectiveness of the proposed method can be further verified. However, for the benchmark large sample datasets, the combination of KHLD and existing algorithms reduces the test performance of these algorithms, which further shows that the method of KHLD in this study is not suitable for large sample set learning and classification.Table 2
Performance comparison on benchmark small sample datasets.
DatasetsMethodsTraining time (s)TestingNo. of kernelsηoηaBloodSVM5.7477.2775.53265KHLD + SVM13.8577.9275.36113,265[a]ELM0.0176.4875.3280KHLD + ELM2.6976.8375.51113,80[a]HSARBF-ELM4.2179.1277.69113,60[a]KHLD + HSARBF-ELM7.3979.8578.14CO-BP3.8572.1371.288KHLD + CO-BP6.8373.8973.34113,8[a]FOGDM-RBF5.6977.1975.3730KHLD + FOGDM-RBF7.8778.3576.92113,30[a]ClimateSVM1.8692.3292.6449KHLD + SVM3.7492.8692.8213,49ELM0.0291.8591.5350KHLD + ELM0.8692.1391.7913,50HSARBF-ELM2.7893.4792.4113,50KHLD + HSARBF-ELM3.8294.2193.13CO-BP1.7392.8092.388KHLD + CO-BP2.5393.6892.7613,8FOGDM-RBF2.1192.2692.078KHLD + FOGDM-RBF3.2093.3992.8213,8HDSVM0.1581.7085.8542KHLD + SVM0.2882.5885.7612,42ELM079.5678.6230KHLD + ELM0.0580.4179.8312,30HSARBF-ELM0.3583.1383.6412,20KHLD + HSARBF-ELM0.6784.3284.39CO-BP0.0880.5880.375KHLD + CO-BP0.2381.5281.2612,5FOGDM-RBF0.1382.7482.1810KHLD + FOGDM-RBF0.2783.5682.9012,10SonarSVM0.1280.8584.9746KHLD + SVM0.2682.5685.3449,46ELM070.3770.0650KHLD + ELM0.1372.9272.7349,50HSARBF-ELM0.6476.7476.0949,40KHLD + HSARBF-ELM0.9579.5980.31CO-BP0.1468.3266.865KHLD + CO-BP0.3670.3169.2549,5FOGDM-RBF0.2773.6372.5816KHLD + FOGDM-RBF0.6274.8274.2949,16SHSVM0.1870.0572.4178KHLD + SVM0.7672.5973.6114,78ELM066.2465.5650KHLD + ELM0.2367.9266.7214,50HSARBF-ELM0.4167.5865.8414,40KHLD + HSARBF-ELM0.7171.2470.38CO-BP0.1271.2670.687KHLD + CO-BP0.3873.1872.8014,7FOGDM-RBF0.4568.6167.3712KHLD + FOGDM-RBF0.8270.6570.1314,12ISSVM7.4690.56—96KHLD + SVM20.5391.84—32,96ELM0.0390.31—100KHLD + ELM1.6890.82—32,100HSARBF-ELM5.1792.23—32,80KHLD + HSARBF-ELM7.2393.17—CO-BP1.5290.74—8KHLD + CO-BP2.6791.85—32,8FOGDM-RBF4.6389.58—36KHLD + FOGDM-RBF6.3990.54—32,36ForestsSVM1.8672.1974.51125KHLD + SVM2.9473.6074.7851,125ELM0.0168.5667.6360KHLD + ELM1.4969.3268.5251,60HSARBF-ELM1.8369.9369.3751,50KHLD + HSARBF-ELM3.3571.6571.32CO-BP1.2763.7962.239KHLD + CO-BP1.8564.1662.5851,9FOGDM-RBF2.3269.2468.8730KHLD + FOGDM-RBF3.9470.6270.4251,30[a]The number of generated RBF kernels and the number of kernels/support vectors in each classifier.Table 3
Performance comparison on benchmark large sample datasets.
DatasetsMethodsTraining time (s)TestingNo. of kernelsηoηaSvmguide1ELM0.1590.5290.36200KHLD + ELM32.5389.8189.73442,200HSARBF-ELM387.5291.7691.34442,160KHLD + HSARBF-ELM423.6491.2490.81CO-BP18.5490.4190.1330KHLD + CO-BP55.3789.3889.22442,30FOGDM-RBF28.5392.3492.1870KHLD + FOGDM-RBF65.7091.8291.56442,70WiltELM0.0662.6360.40100KHLD + ELM37.9161.8259.84173,100HSARBF-ELM358.6164.7363.93173,80KHLD + HSARBF-ELM387.4264.1763.49CO-BP38.5260.2859.5230KHLD + CO-BP78.6559.5658.72173,30FOGDM-RBF24.7362.8562.3760KHLD + FOGDM-RBF63.8562.3962.13173,60


### 3.3. Discussion of KHLD

In this study, under the premise of the given initial kernel widthσ, according to the optimization method of RBF kernel parameters in HSARBF-ELM, the parametersK,μk,σkk=1K of KHLD are automatically generated according to the distribution of sample space, where σkis chosen in σ−0.2≤σk≤σ. When each kernel parameter is established, the main parameter affecting KHLD is θk, which determines the number of samples generated in the kernel. θk is chosen in the σk/5,σk/10,σk/20. Thus, we mainly discuss the influence of parameters of σ and θk on KHLD. Figure 13 shows the stress test when KHLD is combined with HSARBF-ELM in the Climate high-dimensional dataset. In general, when σ and θk are in a certain range, the combination of KHLD and HSARBF-ELM can improve the network performance of HSARBF-ELM. When σ is too low, for example, σ is set as 0.1 or 0.2, the classification performance of KHLD combined with HSARBF-ELM is poor. The main reason is that the generated kernel cannot effectively cover the sample space, so the effectiveness of the kernel cannot be guaranteed, which leads to the performance degradation of the proposed method. When σ is too large or θk is too small, the probability of overlapping samples in the generated kernel will increase, which leads to the performance degradation of the proposed method.Figure 13
Stress testing of the proposed method on climate dataset. (a)σ, number of kernels. (b) σ, overall testing accuracy.
(a)(b)From experiments on multiple datasets, the method of KHLD improves the problem of network generalization performance degradation when the sample size is too small or the sample space distribution is too sparse.However, when the number of training samples is sufficient or the spatial distribution of training samples is dense, the network performance of the proposed method shows a certain degree of decline compared with the direct training of the classifier. This situation shows that when the constructed kernel can be effectively represented by the existing training samples, the generated samples in the kernels are equivalent to increasing the noise samples, which leads to the redundancy of network training and is not conducive to the improvement of the boundary partition surface. Thus, the proposed method is not suitable for classification problems with sufficient number of training samples or dense spatial distribution of samples. In the selection of parameters, the kernel width should be chosen so that it is not too small or too large. If the kernel width is too small, the validity of the established kernel may not be guaranteed, which makes the method of KHLD ineffective to a certain extent. If the kernel width is too large, the overlapping degree between the samples generated in the kernel and the heterogeneous samples increases, which also leads to the performance degradation of the proposed method.


## 3.1. Artificial Datasets: DM and CC

In this section, two artificial datasets are used to verify the graphical and intuitive characteristics of KHLD. In the phase of classification performance comparison, KHLD is combined with HSARBF-ELM and compared with HSARBF-ELM. Figures4(a)–4(d) give a comparison of the learning and classification effects based on the original training set and the KHLD under the DM dataset. It can be seen that the RBF kernel generated in HSARBF-ELM can effectively cover the sample space. The combination of KHLD and HSARBF-ELM can fill the training sample space and effectively improve the classification performance of the method of HSARBF-ELM.Figure 4
The learning and classification comparison of the HSARBF-ELM network classifier based on the original training set and kernel holistic learning and division in the DM dataset, where the number of original training sets is 100 and the initial kernel width is 0.1. (a) Learning the original training set to generate different RBF kernels. (b) Further learning and screening on the basis of each RBF kernel to generate new sample vectors. (c) Classification effect obtained by learning the parameters of the classifier using the original training set. (d) Classification effect obtained by merging the original sample set with the newly generated sample set and learning the classifier parameters.
(a)(b)(c)(d)Figures5 and 6 show the optimization effect of the kernels and the samples generated in each kernel after adjusting parameters σ and θk, respectively, which shows that the adjustment of parameters σ and θk has good adaptability to the sample space on DM dataset.Figure 5
The effect comparison of kernel holistic learning when the number of original training samples is 100 and the initial kernel width takes different parameters. (a)σ=0.1. (b) σ=0.3. (c) σ=0.4.
(a)(b)(c)Figure 6
Comparison of the kernel holistic learning effect under different Parzen window width parameters. (a)θk=σk/5. (b) θk=σk/10. (c) θk=σk/20.
(a)(b)(c)Figure7 compares the number of samples generated and the classification accuracy under different initial training sets, where θk=δk/10. It can be seen that, under the condition of small number of training samples, when the initial kernel width is too small, the established kernel cannot effectively cover the sample space, which leads to the decline of network generalization performance; when the kernel width is large and the number of training samples is sufficient, the performance of the proposed method will also show a certain degree of decline, which shows that the method of KHLD has certain restrictions on the number of training samples and the selection of kernel width parameter.Figure 7
Comparison of the number of training samples and the classification accuracy under different initial training sets in the DM dataset. (a) Number of original training samples, number of generated training samples. (b) Number of original training samples, overall testing accuracy (%).
(a)(b)Figure8 shows the learning and classification comparison of the HASRBF-ELM network classifier based on the original training set and KHLD under the CC dataset. It can be seen that when the generated kernels of different categories overlap each other seriously, the proposed method can still generate new samples in different kernels and improve the classification performance of the original HASRBF-ELM network classifier, which shows the effectiveness of KHLD method for complex classification problems.Figure 8
Sample generation of the original training set under the CC dataset and the comparison of the classification results on the test set. (a) Learning the original training set to generate the subkernel. (b) Further learning and screening to generate new sample vectors. (c) Classification results obtained by using the original training set to learn the classifier parameters. (d) Classifying the original training set with the newly generated training set and then learning the classifier parameters.
(a)(b)(c)(d)Figure9 shows the learning effect of the proposed method on the training set as the initial kernel width varies. By changing different kernel width parameters, the method of KHLD can optimize the selection of samples in each kernel. When the kernel width increases, the generated kernels may cover the heterogeneous samples, resulting in the increase of the overlapping of the samples of different pattern categories in the kernel.Figure 9
Comparison of the effect of sample generation when the number of original training samples is 100 and the initial kernel width is different. (a)σ=0.1. (b) σ=0.2. (c) σ=0.3.
(a)(b)(c)Figure10 further shows the number of generated training samples and the performance comparison of the classification accuracy under different initial training sets, where θk=δk/5. When the width parameters of the RBF kernels are in a certain range, the method of KHLD has a good classification effect. Similar to Figure 7(b), when the initial kernel width is too small, the testing accuracy of the proposed method is greatly reduced, which means that the failure of the initial RBF kernel may invalidate the method of kernel holistic learning and further deteriorate the final classification performance. Thus, it is necessary to avoid such a situation. This condition is also a restrictive condition for KHLD in this study.Figure 10
Comparison of the classification performance and the number of generated training samples on the CC dataset. (a) Number of original training samples, number of generated training samples. (b) Number of original training samples, overall testing accuracy (%).
(a)(b)Figures11 and 12 show that the combination of KHLD and HSARBF-ELM increases the training time. However, the proposed method improves the network classification performance of HSARBF-ELM, especially when the number of training samples is small. When the number of training samples is sufficient, the proposed method will reduce the performance of HSARBF-ELM to a certain extent, which shows that this method of KHLD is suitable for the situation with less training samples or sparse spatial distribution of samples.Figure 11
Performance comparison of the proposed method and HSARBF-ELM in the DM dataset. (a) Number of original training samples, training time. (b) Number of original training samples, overall testing accuracy (%).
(a)(b)Figure 12
Performance comparison of the proposed method and HSARBF-ELM in the CC dataset. (a) Number of original training samples, training time. (b) Number of original training samples, overall testing accuracy (%).
(a)(b)


## 3.2. UCI Benchmark Datasets

Tables2 and 3 give the comparisons of the classification performance of the proposed method and other learning algorithms under the benchmark sample datasets. It can be seen that, in high-dimensional small sample datasets, the combination of KHLD and other classification algorithms increases the training time. Although the testing results of different classification algorithms on different datasets are different, the combination of KHLD and these classification algorithms can improve the testing accuracy of these algorithms to varying degrees. As an auxiliary method, KHLD is an effective method when the spatial distribution of samples is sparse. The effectiveness of the proposed method can be further verified. However, for the benchmark large sample datasets, the combination of KHLD and existing algorithms reduces the test performance of these algorithms, which further shows that the method of KHLD in this study is not suitable for large sample set learning and classification.Table 2
Performance comparison on benchmark small sample datasets.
DatasetsMethodsTraining time (s)TestingNo. of kernelsηoηaBloodSVM5.7477.2775.53265KHLD + SVM13.8577.9275.36113,265[a]ELM0.0176.4875.3280KHLD + ELM2.6976.8375.51113,80[a]HSARBF-ELM4.2179.1277.69113,60[a]KHLD + HSARBF-ELM7.3979.8578.14CO-BP3.8572.1371.288KHLD + CO-BP6.8373.8973.34113,8[a]FOGDM-RBF5.6977.1975.3730KHLD + FOGDM-RBF7.8778.3576.92113,30[a]ClimateSVM1.8692.3292.6449KHLD + SVM3.7492.8692.8213,49ELM0.0291.8591.5350KHLD + ELM0.8692.1391.7913,50HSARBF-ELM2.7893.4792.4113,50KHLD + HSARBF-ELM3.8294.2193.13CO-BP1.7392.8092.388KHLD + CO-BP2.5393.6892.7613,8FOGDM-RBF2.1192.2692.078KHLD + FOGDM-RBF3.2093.3992.8213,8HDSVM0.1581.7085.8542KHLD + SVM0.2882.5885.7612,42ELM079.5678.6230KHLD + ELM0.0580.4179.8312,30HSARBF-ELM0.3583.1383.6412,20KHLD + HSARBF-ELM0.6784.3284.39CO-BP0.0880.5880.375KHLD + CO-BP0.2381.5281.2612,5FOGDM-RBF0.1382.7482.1810KHLD + FOGDM-RBF0.2783.5682.9012,10SonarSVM0.1280.8584.9746KHLD + SVM0.2682.5685.3449,46ELM070.3770.0650KHLD + ELM0.1372.9272.7349,50HSARBF-ELM0.6476.7476.0949,40KHLD + HSARBF-ELM0.9579.5980.31CO-BP0.1468.3266.865KHLD + CO-BP0.3670.3169.2549,5FOGDM-RBF0.2773.6372.5816KHLD + FOGDM-RBF0.6274.8274.2949,16SHSVM0.1870.0572.4178KHLD + SVM0.7672.5973.6114,78ELM066.2465.5650KHLD + ELM0.2367.9266.7214,50HSARBF-ELM0.4167.5865.8414,40KHLD + HSARBF-ELM0.7171.2470.38CO-BP0.1271.2670.687KHLD + CO-BP0.3873.1872.8014,7FOGDM-RBF0.4568.6167.3712KHLD + FOGDM-RBF0.8270.6570.1314,12ISSVM7.4690.56—96KHLD + SVM20.5391.84—32,96ELM0.0390.31—100KHLD + ELM1.6890.82—32,100HSARBF-ELM5.1792.23—32,80KHLD + HSARBF-ELM7.2393.17—CO-BP1.5290.74—8KHLD + CO-BP2.6791.85—32,8FOGDM-RBF4.6389.58—36KHLD + FOGDM-RBF6.3990.54—32,36ForestsSVM1.8672.1974.51125KHLD + SVM2.9473.6074.7851,125ELM0.0168.5667.6360KHLD + ELM1.4969.3268.5251,60HSARBF-ELM1.8369.9369.3751,50KHLD + HSARBF-ELM3.3571.6571.32CO-BP1.2763.7962.239KHLD + CO-BP1.8564.1662.5851,9FOGDM-RBF2.3269.2468.8730KHLD + FOGDM-RBF3.9470.6270.4251,30[a]The number of generated RBF kernels and the number of kernels/support vectors in each classifier.Table 3
Performance comparison on benchmark large sample datasets.
DatasetsMethodsTraining time (s)TestingNo. of kernelsηoηaSvmguide1ELM0.1590.5290.36200KHLD + ELM32.5389.8189.73442,200HSARBF-ELM387.5291.7691.34442,160KHLD + HSARBF-ELM423.6491.2490.81CO-BP18.5490.4190.1330KHLD + CO-BP55.3789.3889.22442,30FOGDM-RBF28.5392.3492.1870KHLD + FOGDM-RBF65.7091.8291.56442,70WiltELM0.0662.6360.40100KHLD + ELM37.9161.8259.84173,100HSARBF-ELM358.6164.7363.93173,80KHLD + HSARBF-ELM387.4264.1763.49CO-BP38.5260.2859.5230KHLD + CO-BP78.6559.5658.72173,30FOGDM-RBF24.7362.8562.3760KHLD + FOGDM-RBF63.8562.3962.13173,60


## 3.3. Discussion of KHLD

In this study, under the premise of the given initial kernel widthσ, according to the optimization method of RBF kernel parameters in HSARBF-ELM, the parametersK,μk,σkk=1K of KHLD are automatically generated according to the distribution of sample space, where σkis chosen in σ−0.2≤σk≤σ. When each kernel parameter is established, the main parameter affecting KHLD is θk, which determines the number of samples generated in the kernel. θk is chosen in the σk/5,σk/10,σk/20. Thus, we mainly discuss the influence of parameters of σ and θk on KHLD. Figure 13 shows the stress test when KHLD is combined with HSARBF-ELM in the Climate high-dimensional dataset. In general, when σ and θk are in a certain range, the combination of KHLD and HSARBF-ELM can improve the network performance of HSARBF-ELM. When σ is too low, for example, σ is set as 0.1 or 0.2, the classification performance of KHLD combined with HSARBF-ELM is poor. The main reason is that the generated kernel cannot effectively cover the sample space, so the effectiveness of the kernel cannot be guaranteed, which leads to the performance degradation of the proposed method. When σ is too large or θk is too small, the probability of overlapping samples in the generated kernel will increase, which leads to the performance degradation of the proposed method.Figure 13
Stress testing of the proposed method on climate dataset. (a)σ, number of kernels. (b) σ, overall testing accuracy.
(a)(b)From experiments on multiple datasets, the method of KHLD improves the problem of network generalization performance degradation when the sample size is too small or the sample space distribution is too sparse.However, when the number of training samples is sufficient or the spatial distribution of training samples is dense, the network performance of the proposed method shows a certain degree of decline compared with the direct training of the classifier. This situation shows that when the constructed kernel can be effectively represented by the existing training samples, the generated samples in the kernels are equivalent to increasing the noise samples, which leads to the redundancy of network training and is not conducive to the improvement of the boundary partition surface. Thus, the proposed method is not suitable for classification problems with sufficient number of training samples or dense spatial distribution of samples. In the selection of parameters, the kernel width should be chosen so that it is not too small or too large. If the kernel width is too small, the validity of the established kernel may not be guaranteed, which makes the method of KHLD ineffective to a certain extent. If the kernel width is too large, the overlapping degree between the samples generated in the kernel and the heterogeneous samples increases, which also leads to the performance degradation of the proposed method.


## 4. Conclusion

An optimized neural network classifier based on KHLD is presented. The established kernels in KHLD are based on the generated RBF kernel parameters in the HSARBF-ELM algorithm. An optimized sample filling and screening method can realize the effective approximation of KHLD in different classification problems. Combining KHLD with other algorithms can effectively improve the network performance of these algorithms, especially when the sample space distribution is sparse. Experiments on artificial datasets and benchmark datasets further verify the effectiveness of our method.One of the main shortcomings of this work is the representation of kernels. In this study, for the convenience of problem description, the representation of the kernel is a regular hypersphere. The proposed method is mainly suitable for the case of sparse spatial distribution of samples but is not suitable for large sample set learning and classification. The establishment and representation of kernel are worthy of further study. Exploring more optimized kernel representation and combining it with KHLD are our future work.



---

*Source: 8857818-2021-02-28.xml*