# Ensemble Learning Based Multiple Kernel Principal Component Analysis for Dimensionality Reduction and Classification of Hyperspectral Imagery

Authors: Hamidullah Binol
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

Classification is one of the most challenging tasks of remotely sensed data processing, particularly for hyperspectral imaging (HSI). Dimension reduction is widely applied as a preprocessing step for classification; however the reduction of dimension using conventional methods may not always guarantee high classification rate. Principal component analysis (PCA) and its nonlinear version kernel PCA (KPCA) are known as traditional dimension reduction algorithms. In a previous work, a variant of KPCA, denoted as Adaptive KPCA (A-KPCA), is suggested to get robust unsupervised feature representation for HSI. The specified technique employs several KPCAs simultaneously to obtain better feature points from each applied KPCA which includes different candidate kernels. Nevertheless, A-KPCA neglects the influence of subkernels employing an unweighted combination. Furthermore, if there is at least one weak kernel in the set of kernels, the classification performance may be reduced significantly. To address these problems, in this paper we propose an Ensemble Learning (EL) based multiple kernel PCA (M-KPCA) strategy. M-KPCA constructs a weighted combination of kernels with high discriminative ability from a predetermined set of base kernels and then extracts features in an unsupervised fashion. The experiments on two different AVIRIS hyperspectral data sets show that the proposed algorithm can achieve a satisfactory feature extraction performance on real data.


---

## Body

## 1. Introduction

Hyperspectral imaging (HSI) provides simultaneously spatial and high resolution spectral data and helps to classify/recognize the materials that are challenging to discriminate with conventional imaging techniques [1]. However, it suffers from the curse of dimensionality. For instance, the curse of dimensionality causes increase in cost of storage, transmission, and processing of hyperspectral images. To overcome such challenges, dimensionality reduction techniques have been applied to hyperspectral data in the existing literature [2]. In general, HSI has spectral redundancy in many spectral channels. For this reason, dimension reduction or compression is possible and even necessary, especially for these bands.Even though there are several dimension reduction approaches in the literature, including manifold learning [3, 4] and tensors [5], principal component analysis (PCA) [6] is the one among the popular techniques [7–9]. PCA is the discrete form of the continuous Karhunen-Loève Transform and it projects the data into a subspace so that the variance retained is maximized and the least square reconstruction error is minimized [10]. Use of PCA for dimensionality reduction in HSI is a computationally suitable approach and it helps preserve the most of the variance of the raw data. Although PCA has some theoretical inadequacies [11, 12] for use on remote sensing data, particularly hyperspectral images [13], the practical applications show that the results obtained using PCA are still competitive for the purpose of classification [14, 15]. The ability of PCA is limited for high-dimensional data since it relies on only second-order statistical information. The nonlinear version of the PCA, denoted as kernel PCA (KPCA), has been proposed to overcome these limitations [16].Since the KPCA involves the higher-order statistics, it provides more information from the original data [17] and so it is employed in many applications including remote sensing data due to its satisfactory performance. In [18], classification performance of an artificial neural network has been demonstrated to outperform the classical approach using kernel principal components. Fauvel et al. [19] showed that the KPCA is better than the classical PCA in terms of classification accuracies. A general overview of feature reduction techniques for classification of hyperspectral images is presented in [9]. They performed comparative experiments between the unsupervised, e.g., PCA and KPCA, and supervised techniques, e.g., double nearest proportion (DNP) [20] and kernel nonparametric weighted feature extraction (KNWFE) [21]. Since the supervised learning techniques generally focus on improving class separability, these methods are expected to produce better results in terms of classification performance. The comparative results with KNWFE indicate that PCA and KPCA are still preferable to reduce dimensionality of hyperspectral images.Fundamentally, KPCA is a version of PCA whose performance is greatly affected by the choice of the kernel and parameters. Namely, the selection of the optimal kernel and parameters is crucial for KPCA to achieve good performance. However, the application results show that no single kernel function can be best for all kinds of machine learning problems [22] and, therefore, learning of optimum kernels over a kernel set is an active research area nowadays [23–27]. Li and Yang presented an ensemble KPCA method with Bayesian inference strategy in [28]. They exploited only Gaussian radial basis function (RBF) with different scale parameters as subkernels. Zhang et al. [29] have developed a method for unsupervised kernel learning in the KPCA, dubbed as A-KPCA, and applied the new method for object recognition problems.The A-KPCA learns the kernels via an unsupervised learning approach. The 1D input vectors, e.g., feature vectors, are transformed into 2D feature matrices by different kernels. Each column of the feature matrix comes from corresponding 1D input vector. Nonlinear feature extraction (FE) is obtained from one set of projective vectors corresponding to the column direction of the feature matrices. The set of projective vectors corresponding the row direction of the 2D feature matrices is utilized for searching optimal kernels combination simultaneously. Despite having superior performance compared to KPCA, the A-KPCA has some critical limitations. Specifically, A-KPCA works completely unsupervised, and it is thus incapable of enhancing the class separability and it has no kernel preselection process. These are the main motivations of our work.In this paper, a novel framework is introduced for hyperspectral FE and classification based on multiple KPCA models with an Ensemble Learning (EL) strategy in a semisupervised manner. EL is a process of combining multiple models, called experts, to set up a strong model for a specific machine learning problem [30]. Strong discriminative ability of individual experts and high diversity among them are required to produce satisfactory models [31, 32]. An acceptable classification performance highly depends on the class separability of features that is directly related to the discriminative ability. Inspired by EL, we extend the A-KPCA method by employing multiple kernels such that subkernels possessing higher discrimination ability are highlighted. The proposed approach, multiple kernel PCA (M-KPCA), learns an ensemble of multiple kernel principal components on an available labeled data set, and the final features are extracted via a weighted combination of all subkernels according to their separability performance. The early purpose of this paper is the utilization of the KPCA and A-KPCA in hyperspectral images and to determine impact of using nonlinear versions of PCA on classification performance. The further contributions and novelties in this paper can be summarized as follows: (1) a novel multikernel PCA strategy is presented by exploiting Ensemble Learning to evaluate and select the kernels; (2) M-KPCA acquires the superior classification results than PCA, KPCA, and A-KPCA by highlighting the subkernels with a class separability based weighting strategy; (3) M-KPCA produces better or competitive classification performance with other popular unsupervised FE methods like locality preserving projections (LPP) [33], random projections (RP) [34], and t-distributed stochastic neighbor embedding (t-SNE) [35]. After FE with all mentioned methods, the popular and robust support vector machines (SVMs) classifier is used for supervised classification. Since SVMs consider samples close to the class boundary, calledsupport vectors, they show great performance even in high-dimensional data with small training samples [36, 37].The paper is outlined as follows. Section2 reviews the related work. In Section 3, the proposed framework of M-KPCA is presented. Next, a series of experiments are carried out on real data sets for verifying our method’s effect in Section 4. Finally, Section 5 concludes this paper.


## 2. Related Work

### 2.1. KPCA Background

The raw data is projected into the feature space by a nonlinear mapping function and the useful information is concentrated into some principal components corresponding to the larger eigenvalues [19]. Define a learning set as xk(k=1,2,…,N), xk∈Rd. Let φ:Rd→RX be a nonlinear mapping from the input space to a high-dimensional feature spaceH. The inner product in feature space is calculated by the kernel function in the original input space:(1)Kxi,xj=φxi⊤φxjwhere the superscript ⊤ represents the transpose operation. Denote Φ=(φ(x1),φ(x2),…,φ(xN)) and φ-=1/N∑k=1Nφ(xk). Assuming φ-=0, i.e., data are centered in H, then the total scatter matrix can be defined as St=ΦΦ⊤. To compute the projective vector v for optimal solution, the KPCA employs the following norm:(2)Jv=∑k=1Nv⊤φxk2=v⊤Stv.Computation of optimal projective vectorv provides solution for the eigenvalue problem: λv=Stv in which λ≥0 and eigenvectors v∈H. Hence, (2) can be rewritten as an equivalent problem:(3)λz=Kz,where K=Φ⊤Φ is the kernel matrix. Solutions of (3) are {z1,z2,…,zc} corresponding to the largest c eigenvalues; then vk=Φzk is the solution vector of (2). The KPCA based FE does not include the nonlinear mapping φ as any kernel method, and it only needs a kernel function in the input space. To obtain better performance with KPCA, the parameters of the kernel are optimized. However, this optimization cannot produce adequate solutions for every application or data sets because of the nature of the kernel itself [22]. To overcome this drawback, an adaptive kernel combination technique is introduced in [29].


### 2.2. Adaptive KPCA (A-KPCA)

As pointed out in Section1, the performance of KPCA is notably affected by the selection of kernels and its parameters. Therefore, it needs some extensions. Let ψi(x)∈Fi,i∈{1,2,…,h} be a set of nonlinear mappings. As mentioned in Section 2.1, the inner products in Fi are described as the kernels. Using definition of ψi, ψ^i:x→ψ^i(x)=(0⊤…0⊤︸1…i-1,ψi(x)⊤,0⊤…0⊤︸i+1…h)⊤∈F can be written. In this equation, F is the Hilbert space as the direct sum of Fi and the inner product in F can be defined as(4)ψ^ix⊤ψ^jy=0i≠j,(5)ψ^ix⊤ψ^iy=ψix⊤ψiy=Kix,y.To construct a 2D feature matrix, a sample of learning setxk is transformed to high-dimensional feature space and then ψk=(ψ^1(xk),ψ^2(xk),…,ψ^h(xk)) is obtained. Here, each column of ψk corresponds to a nonlinear mapping generated by ψ^i’s. Thus, vector-based data is converted to matrix based format. Assuming ψk’s have zero means, i.e., ψ-=0, ψ-=1/N∑i=1Nψi can be written. Equation (6) includes generated ψ1,ψ2,…,ψN feature vectors. Appropriate M and N matrices must be determined to optimize(6)JM,N=∑k=1NM⊤ψkNF2,where M=(m1,m2,…,mt) are projective vectors corresponding to columns of ψk while N=(n1,n2,…,np) corresponding to rows of ψk. The purpose of M is to extract features, while the purpose of N is kernel selection. In other words, the unsupervised kernel learning and nonlinear FE are simultaneously realized according to projective vectors which are included in M and N. ∙F is theFrobenius norm of matrix, i.e., AF2≔trAA⊤=∑i,jAij2, where tr denotes the trace of a matrix. It can be defined as M=ψL, where ψ=(ψ1,ψ2,…,ψN). Since the size of original ψ is very large, i.e., (h×N), ψ~=(∑ihψ^i(x1),∑ihψ^i(x2),…,∑ihψ^i(xN)) can be written instead of ψ. Hence, M=ψ~L,L∈RN×t is obtained. These calculations allow us to rewrite (6) as (7):(7)JL,N=∑k=1NL⊤ψ~⊤ψkNF2,=∑k=1NL⊤KkNF2.where the constrains of (7) are L∈RN×t,L⊤L=I and N∈Rh×p,N⊤N=I. Here Kk=ψ~⊤ψ is N×h sized kernel matrix and it is constructed as follows:(8)Kki,j=Kjxi,xk.To solve this optimization problem, inspired by Ye’s work [38], an iterative procedure is presented by the following theorem [29].Theorem 1.
LetL and N be the optimal solution to (7): then (i) t eigenvectors corresponding to the largest t eigenvalues of the matrix AL=∑k=1NKkRR⊤Kk⊤ form L for a given N; (ii) p eigenvectors corresponding to the largest p eigenvalues of the matrix AN=∑k=1NKkLL⊤Kk⊤ create N for a given L.After computingL and N, these matrices can be used to extract the nonlinear features for a test instance z∈Rd. Kernel matrix Ktest(i,j)=Kj(xi,z) is constructed and then projected according to C=L⊤KtestN, so the nonlinear features are contained in C. The A-KPCA method is given in Algorithm 1.Algorithm 1:A-KPCA Algorithm.
Input: Given training set xkk=1,2,…,N. (a) Create the kernel matrix Kk for each xk. (b) Get initial N0 and i←1. (c) For given Ni-1, calculate the t eigenvectors of AL corresponding to the largest t eigenvalues. (d) For given Li, calculate the p eigenvectors of AN corresponding to the largest p eigenvalues. (e) i←i+1, goto step (c) until convergence. Output: L and N.


## 2.1. KPCA Background

The raw data is projected into the feature space by a nonlinear mapping function and the useful information is concentrated into some principal components corresponding to the larger eigenvalues [19]. Define a learning set as xk(k=1,2,…,N), xk∈Rd. Let φ:Rd→RX be a nonlinear mapping from the input space to a high-dimensional feature spaceH. The inner product in feature space is calculated by the kernel function in the original input space:(1)Kxi,xj=φxi⊤φxjwhere the superscript ⊤ represents the transpose operation. Denote Φ=(φ(x1),φ(x2),…,φ(xN)) and φ-=1/N∑k=1Nφ(xk). Assuming φ-=0, i.e., data are centered in H, then the total scatter matrix can be defined as St=ΦΦ⊤. To compute the projective vector v for optimal solution, the KPCA employs the following norm:(2)Jv=∑k=1Nv⊤φxk2=v⊤Stv.Computation of optimal projective vectorv provides solution for the eigenvalue problem: λv=Stv in which λ≥0 and eigenvectors v∈H. Hence, (2) can be rewritten as an equivalent problem:(3)λz=Kz,where K=Φ⊤Φ is the kernel matrix. Solutions of (3) are {z1,z2,…,zc} corresponding to the largest c eigenvalues; then vk=Φzk is the solution vector of (2). The KPCA based FE does not include the nonlinear mapping φ as any kernel method, and it only needs a kernel function in the input space. To obtain better performance with KPCA, the parameters of the kernel are optimized. However, this optimization cannot produce adequate solutions for every application or data sets because of the nature of the kernel itself [22]. To overcome this drawback, an adaptive kernel combination technique is introduced in [29].


## 2.2. Adaptive KPCA (A-KPCA)

As pointed out in Section1, the performance of KPCA is notably affected by the selection of kernels and its parameters. Therefore, it needs some extensions. Let ψi(x)∈Fi,i∈{1,2,…,h} be a set of nonlinear mappings. As mentioned in Section 2.1, the inner products in Fi are described as the kernels. Using definition of ψi, ψ^i:x→ψ^i(x)=(0⊤…0⊤︸1…i-1,ψi(x)⊤,0⊤…0⊤︸i+1…h)⊤∈F can be written. In this equation, F is the Hilbert space as the direct sum of Fi and the inner product in F can be defined as(4)ψ^ix⊤ψ^jy=0i≠j,(5)ψ^ix⊤ψ^iy=ψix⊤ψiy=Kix,y.To construct a 2D feature matrix, a sample of learning setxk is transformed to high-dimensional feature space and then ψk=(ψ^1(xk),ψ^2(xk),…,ψ^h(xk)) is obtained. Here, each column of ψk corresponds to a nonlinear mapping generated by ψ^i’s. Thus, vector-based data is converted to matrix based format. Assuming ψk’s have zero means, i.e., ψ-=0, ψ-=1/N∑i=1Nψi can be written. Equation (6) includes generated ψ1,ψ2,…,ψN feature vectors. Appropriate M and N matrices must be determined to optimize(6)JM,N=∑k=1NM⊤ψkNF2,where M=(m1,m2,…,mt) are projective vectors corresponding to columns of ψk while N=(n1,n2,…,np) corresponding to rows of ψk. The purpose of M is to extract features, while the purpose of N is kernel selection. In other words, the unsupervised kernel learning and nonlinear FE are simultaneously realized according to projective vectors which are included in M and N. ∙F is theFrobenius norm of matrix, i.e., AF2≔trAA⊤=∑i,jAij2, where tr denotes the trace of a matrix. It can be defined as M=ψL, where ψ=(ψ1,ψ2,…,ψN). Since the size of original ψ is very large, i.e., (h×N), ψ~=(∑ihψ^i(x1),∑ihψ^i(x2),…,∑ihψ^i(xN)) can be written instead of ψ. Hence, M=ψ~L,L∈RN×t is obtained. These calculations allow us to rewrite (6) as (7):(7)JL,N=∑k=1NL⊤ψ~⊤ψkNF2,=∑k=1NL⊤KkNF2.where the constrains of (7) are L∈RN×t,L⊤L=I and N∈Rh×p,N⊤N=I. Here Kk=ψ~⊤ψ is N×h sized kernel matrix and it is constructed as follows:(8)Kki,j=Kjxi,xk.To solve this optimization problem, inspired by Ye’s work [38], an iterative procedure is presented by the following theorem [29].Theorem 1.
LetL and N be the optimal solution to (7): then (i) t eigenvectors corresponding to the largest t eigenvalues of the matrix AL=∑k=1NKkRR⊤Kk⊤ form L for a given N; (ii) p eigenvectors corresponding to the largest p eigenvalues of the matrix AN=∑k=1NKkLL⊤Kk⊤ create N for a given L.After computingL and N, these matrices can be used to extract the nonlinear features for a test instance z∈Rd. Kernel matrix Ktest(i,j)=Kj(xi,z) is constructed and then projected according to C=L⊤KtestN, so the nonlinear features are contained in C. The A-KPCA method is given in Algorithm 1.Algorithm 1:A-KPCA Algorithm.
Input: Given training set xkk=1,2,…,N. (a) Create the kernel matrix Kk for each xk. (b) Get initial N0 and i←1. (c) For given Ni-1, calculate the t eigenvectors of AL corresponding to the largest t eigenvalues. (d) For given Li, calculate the p eigenvectors of AN corresponding to the largest p eigenvalues. (e) i←i+1, goto step (c) until convergence. Output: L and N.


## 3. Multiple Kernel PCA (M-KPCA)

In Section2, we have demonstrated that A-KPCA manipulates more than one subkernels. A mapping rule transforms input data samples into corresponding Reproducing Kernel Hilbert Space. Each kernel thus acquires a particular type of information from a given data set, thereby providing a partial description of view data. The value of this specific information may vary according to different machine learning tasks such as classification, clustering, dimensionality reduction, etc. For instance, in a classification problem, high discrimination ability of kernels yields the better results. Hence, we add this capability to A-KPCA with ideas of EL. Our proposed technique learns new representation for a hyperspectral image exploiting all available training data. It is thus independent of the classifier.As seen in formulation (8) and Theorem 1, there are not any coefficients to quantify the contribution of subkernels in classification. In other words, the A-KPCA utilizes the unweighted summation. Nevertheless, the discriminative ability of kernels in FE plays significant role for the separability of the classifier. If we add a weighting coefficient on right side of (8), then it becomes(9)Kki,j=αjKjxi,xk.The discriminative ability of a kernel can be measured by an ideal kernel in a given classification task. Cristianini et al. [23] introduced a measure of similarity between two arbitrary kernels or between a kernel and an ideal kernel called kernel alignment (KA). The alignment between two regular kernels is given as(10)AK1,K2=K1,K2FK1,K1FK2,K2Fwhere theFrobenius product .,.F of two Gram matrices A and B is defined as A,BF=tr(AB) [23, 39]. This measure can be viewed as the cosine of the angle between K1 and K2, so it fluctuates between -1,1 for arbitrary matrices. However, since we consider only positive semidefinite Gram matrices in KA, the score is lower bounded by zero. The alignment can also be adopted to capture the degree of agreement between a kernel and the target label matrix, also considered asideal kernel. A larger value of KA indicates the higher discriminative ability and it is one of the main strengths for a subclassifier such that they improve the ensemble effect in an EL strategy [40, 41]. An idealized kernel for a binary classification problem can be composed of the dot product of target labels, i.e., yy⊤, and the alignment between a kernel and the ideal kernel is written as(11)AK,yy⊤=K,yy⊤FKFyy⊤F.Our goal is to construct an A-KPCA based algorithm which has improved separability of multiclass patterns. Here, kernel class separability (KCS) measure based on scatter matrix is employed to measure the class separability of training samples in feature space. The KSC is a general form of KA and it can be written in the form [42]:(12)Jφ=trSBφtrSWφwhere SBφ and SWφ, respectively, stand for between-class scatter matrix and within-class scatter matrix in kernel space and the traces of them are obtained as(13)trSBφ=∑i=1cnimiφ-mφ⊤miφ-mφtrSWφ=∑i=1c∑j=1niφxij-miφ⊤φxij-miφwhere ni(i=1,…,c) denotes the number of training samples in the ith class, N=∑i=1cni, and xij is the jth sample in the related class. mi and m are the mean vector for ith class and the mean vector for all training samples, respectively. φ is the mapping function from the input space to the feature space as described in the beginning of Section 2.1. A larger value of Jφ signifies superior class separability in the training set. A maximization problem may thus be created to obtain optimal kernels and their parameters or eliminate weak kernels [43], but, in this paper, we directly exploit the value of (12) as the measure of discriminability; hence(14)αj=Jφj,where φj(x)⊤φj(y)=Kj(x,y).After all, we extend the noniterative A-KPCA algorithm using kernel class separability measure with a semisupervised strategy. The proposed noniterative M-KPCA technique is given in Algorithm2.Algorithm 2:M-KPCA Algorithm.
Input: Given training set xkk=1,2,…,N with labels yk∈1,2,…,c. (a) Obtain αj’s for corresponding pre-selected kernels using Eqs. (12) and (13). (b) Create the kernel matrix Kk for each xk as in Eq. (9). (c) Calculate the eigenvectors vi and eigenvalues λi of AL=∑k=1NKkKk⊤. Sort the eigenvectors according to the decreasing order of λi and select first t eigenvectors v1,…,vt. (d) Calculate the eigenvectors ri and eigenvalues δi of AN=∑k=1NKk⊤Kk. Sort the eigenvectors according to the decreasing order of δi and select first p eigenvectors r1,…,rp. (e) The final subspaces are L=v1,…,vt and N=r1,…,rp. Output: L and N.


## 4. Experiments

In this section, we investigate the performance of the proposed M-KPCA algorithm compared with a number of conventional and state-of-the-art techniques on two Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) hyperspectral data sets. Our experiments are conducted on a machine with an Intel Core i5-2410M CPU at 2.30GHz and 8GB DDR-III RAM.


### 4.1. Data Sets and Experimental Setup

The first set is an airborne remote sensing data captured by the AVIRIS sensor over northwest Indiana on June 12, 1992. Indian Pines data has 16 labeled classes and 145 lines/scene and 145 pixels/line. Originally, the scene has 220 spectral bands (10 nm spectral bandwidth from 0.4 to 2.5μm); after discarding the water absorption and noise bands, based on [44, 45], only 159 bands were used in the experiments.Airborne hyperspectral data is acquired by AVIRIS sensor at 18 m spatial resolution over Kennedy Space Center (KSC) during March 1996 and has been employed as a second data source. Noisy bands and water absorption bands are removed. The remaining of the HSI data has 176 bands for 13 wetland and upland classes. Figure1 shows the RGB image of Indian Pines and false color image of the KSC. Table 1 lists the summary of the data sets in our experiments. All samples in each data are adjusted in the range [0,1], as suggested in [46].Table 1
Information for Indian Pines and KSC data sets.
No Indian Pines Samples KSC Samples Class name Train Test Class name Train Test 1 Alfalfa 14 40 Scrub 190 571 2 Corn-no till 359 1075 Willow swamp 61 182 3 Corn–min till 209 625 Cabbage hamm 64 192 4 Corn 59 175 Cabbage palm 63 189 5 Grass-pasture 124 373 Slash pine 40 121 6 Grass-trees 187 560 Oak 57 172 7 Grass-past. moved 6 20 Hardwood swamp 264 791 8 Hay-windrowed 122 367 Graminoid marsh 108 323 9 Oats 5 15 Spartina marsh 130 390 10 Soybean-no till 242 726 Cattail marsh 101 303 11 Soybean-min till 617 1851 Salt marsh 105 314 12 Soybean-clean till 154 460 Mud flats 126 377 13 Wheat 53 159 Water 232 695 14 Woods 324 970 — 15 Bldg-Grass-Tree 95 285 — 16 Stone-steel towers 24 71 —  Total  2594 7772  1541 3670Figure 1
AVIRIS data sets: (a) RGB image of Indian Pines obtained from default bands{R:24,G:14,B:8} and (b) false color composite image of KSC obtained from bands {R:40,G:29,B:20}.
(a) (b)The PCA and KPCAs are implemented using the SIMFEAT toolbox [47]. In each experiment, a single kernel is selected for the KPCA. In addition to Gaussian radial basis function (RBF) kernel which is formulated as K(x,y)=exp⁡(-x-y2/2σ2), we have employed three more kernels (see Table 2). Before solving the eigenvalue problem, the parameter σ∈R+ in the RBF, Laplacian and Cauchy kernels should be selected or optimized. Unless otherwise stated the kernel parameter is set to(15)σ=∑Nxk-x-2N,where x- is the centroid of the total N training data [48]. The kernel parameter σ is nonoptimized and same for each exploited kernel given in Table 2. However, the aim of combination of nonoptimized kernels is to yield a better FE technique for classification.Table 2
Employed kernel functions.
Kernels Formula RBF K R B F ( x , y ) = exp ⁡ - x - y 2 2 σ 2  Laplacian K L a p x , y = exp ⁡ - x - y σ  Cauchy K C a u ( x , y ) = 1 1 + x - y 2 / σ 2  Histogram Intersection (HIST) K H I S T x , y =  ∑ i = 1 N min ⁡ x i , y iThe valuable parts of the obtained cumulative eigenvalues after eigen-decomposition for each method are shown in Figure2. According to the cumulative eigenvalues of PCA, two principal components reach 99% of total variance for Indian Pines case. Nevertheless, in KSC case, three principal components are needed to reach 99% of information. According to these results, the new dimensions of Indian Pines and KSC for classification experiment are, respectively, defined as 2 and 3. However, hyperspectral information cannot be represented utilizing only the second-order statistics as it is pointed out in Section 1. From Figure 2, it can be derived that more kernel principal components (KPCs) are needed to realize the same amount of variance as for PCA. Note that the total number of components with PCA is equal to the number of bands, i.e., 159 for Indian Pines, while it is equal for KPCA to the size of the number of training samples, i.e., 2594, which is significantly higher. For the Indian Pines data set, the first 11, 51, 31, and 33 KPCs are needed to accomplish 99% of the cumulative variance with RBF, Laplacian, Cauchy, and histogram intersection (HIST) kernels, respectively. We observe that 8 KPCs are needed with the RBF, 56 with the Laplacian, 18 with the Cauchy, and 55 with the HIST kernel to achieve same amount of information considering to KSC results. In the case of A-KPCA and M-KPCA,p is set to 1 for kernel selection. For the Indian Pines data set, 35 adaptive KPCs and 20 multiple KPCs contain 99% of information and only 14 adaptive KPCs and 12 multiple KPCs for the KSC.Figure 2
Percentages of cumulative variances of the initial data retained in the components of the new representation for the two data sets: (left to right) first row: PCA andKPCARBF; second row: KPCALap and KPCACau; third row: KPCAHIST and A-KPCA; bottom: M-KPCA.In order to demonstrate the first principal components (PCs) more efficiently, a subimage of size 100 × 100 in the KSC hyperspectral cube is selected. The first PCs for all of the methods are depicted in Figure3.Figure 3
(a) False color image of KSC subscene and most informative principal components (PCs) for the subimage of KSC data: (b) 1st PC; (c) 1st KPCRBF; (d) 1st KPCLap; (e) 1st KPCCau; (f) 1st KPCHIST; (g) 1st A-KPC; (h) 1st M-KPC.
(a)
Original (b)
1st PC (c)
1st KPCRBF (d)
1st KPCLap (e)
1st KPCCau (f)
1st KPCHIST (g)
1st A-KPC (h)
1st M-KPCAfter FE, SVM classifier has been employed for classification. For nonlinear SVMs, we have used the RBF kernel which is formulated in Table2. The classification experiments and the optimization of parameters, C and σ, of SVMs are achieved using LIBSVM [49] with 5-fold cross validation technique. Since SVMs are designed to solve binary problems, various approaches have been proposed for multiclass situations such as remote sensing applications. The most popular approaches for multiclass classification are one-against-all (1AA) and one-against-one (1A1). In this paper, we have applied the 1AA strategy for each class. Each test sample is finally labeled as the class whose output score is maximum.Finally, we compare the proposed M-KPCA algorithm against five state-of-the-art dimension reduction algorithms, i.e., linear discriminant analysis (LDA) [50], LPP, probabilistic PCA (pPCA) [51], RP, and t-SNE. LDA, LPP, pPCA, and t-SNE are implemented using the MATLAB toolbox [52] for dimensionality reduction, and RP algorithm is designed based on Wang’s work [53].


### 4.2. Comparison with KPCA and A-KPCA

The original data sets, termed as raw, are also classified for comparisons. Tables3 and 4 compare the performance of all models numerically (class accuracies and overall accuracy (OA) in percentages) and statistically (kappa test) for the Indian Pines and the KSC data sets, respectively.Table 3
Overall accuracy, OA (%), and Kappa statistic,κ, of the Indian Pines data set with different models. The best scores for each class are highlighted in bold face font, and the second best is underlined.
Feature Raw PCA KPCARBF KPCALap KPCACau KPCAHIST A-KPCA M-KPCA  # of features 159 2 11 51 31 33 35 20  SVM C 430.5 279.17 534.67 663.98 663.98 663.98 603.46 346.67 params σ 2.18 15.22 13.37 13.28 13.27 13.28 16.59 4.74  1 87.04 33.34 88.89 74.07 77.78 90.74 90.74 94.44 2 88.77 40.59 86.82 75.80 82.08 83.19 89.82 89.96 3 86.09 06.95 86.81 65.47 82.61 78.30 88.49 88.60 4 92.31 35.90 91.03 68.38 81.62 82.48 91.45 96.58 5 98.19 58.95 97.79 88.73 95.77 90.74 97.59 98.59 6 98.93 91.43 98.39 97.59 97.99 96.52 99.06 99.33 7 92.31 30.77 92.31 88.46 92.31 84.62 92.31 96.15 8 99.39 95.50 98.98 97.75 98.98 97.34 99.39 99.59 9 90.00 00.00 95.00 90.00 85.00 90.00 95.00 95.00 10 89.67 66.32 91.12 80.58 90.50 88.74 91.22 91.84 11 91.90 73.30 93.27 86.95 91.13 89.55 93.11 94.25 12 94.30 21.01 95.11 80.78 90.88 88.27 95.44 95.60 13 99.06 73.58 99.06 98.11 98.58 97.64 99.06 99.53 14 97.30 86.79 98.07 96.91 97.99 96.52 97.84 97.60 15 80.00 15.53 77.11 53.42 70.26 58.95 82.63 88.95 16 92.63 90.53 94.74 94.74 91.58 90.53 90.53 94.74  OA 92.47 59.78 92.69 83.92 90.16 88.30 93.44 94.28 κ 0.9130 0.5399 0.9210 0.8215 0.8967 0.8749 0.9317 0.9348Table 4
Overall accuracy, OA (%), and Kappa statistic,κ, of the KSC data set with different models. The best scores for each class are highlighted in bold face font, and the second best is underlined.
Feature Raw PCA KPCARBF KPCALap KPCACau KPCAHIST A-KPCA M-KPCA  # of features 176 3 8 56 18 55 14 12  SVM C 534.66 279.17 430.54 663.98 181.02 346.69 117.38 165.69 params σ 3.77 9.87 34.95 5.58 10.69 5.58 5.61 5.01  1 97.50 92.64 97.50 95.66 96.58 96.32 98.03 98.29 2 94.24 84.77 93.83 90.12 91.77 91.36 94.24 95.47 3 93.75 89.84 93.36 91.80 93.75 93.75 93.36 97.26 4 78.17 49.21 80.16 71.43 78.17 75.40 81.35 93.25 5 77.64 56.52 60.87 55.28 61.49 57.76 78.88 88.82 6 75.11 50.66 58.08 55.02 75.11 58.95 82.53 86.90 7 93.34 73.33 89.52 91.43 90.48 91.43 94.29 95.24 8 96.29 78.65 94.20 90.26 95.13 91.18 96.75 96.28 9 99.04 97.50 98.65 96.73 98.65 97.50 99.23 99.61 10 100 91.58 99.75 97.77 99.26 97.77 100 99.26 11 99.05 97.85 99.05 98.09 99.05 98.81 99.05 99.28 12 99.06 81.11 96.82 95.23 97.42 95.63 99.01 99.20 13 100 99.89 100 99.89 99.89 99.89 99.89 100  OA 95.51 86.53 93.78 91.65 94.34 92.59 96.14 97.52 κ 0.9522 0.8493 0.9347 0.9106 0.9407 0.9213 0.9601 0.9724Inspection of Table3 reveals A-KPCA outperforms PCA and all the four KPCAs. Further analysis shows that the KPCA performs significantly better than the conventional PCA. Regarding the OAs, it is clear that the M-KPCA based classification produces more accurate results when compared to the A-KPCA based classification. RBF kernel gives the best results for KPCA among the other kernel functions as seen in Table 3.The results for the KSC data set are reported in Table4. Regarding the PCA and KPCA results, FE does not improve the accuracies significantly. The comparison between KPCA and PCA shows that KPCA performs better than the PCA in terms of classification accuracies. Moreover, classification of the A-KPCA features is more precise that the one yielded employing the all KPCs. As with the previous experiment, the best results are obtained with the M-KPCA. Figures 4 and 5 represent the available labeled scenes and classification maps of all models for the Indian Pines and KSC data sets, respectively.Figure 4
Classification results obtained with the Indian Pines: (a) labeled scene and classification maps using the (b) raw data; (c) PCA; (d) KPCARBF; (e) KPCALap; (f) KPCACau; (g) KPCAHIST; (h) A-KPCA; (i) M-KPCA. The classification was done by SVM with an RBF kernel.
(a)
Labeled scene (b)
Raw (c)
PCA (d)
KPCARBF (e)
KPCALap (f)
KPCACau (g)
KPCAHIST (h)
A-KPCA (i)
M-KPCAFigure 5
Classification results obtained with the KSC: (a) labeled scene and classification maps using the (b) raw data; (c) PCA; (d) KPCARBF; (e) KPCALap; (f) KPCACau; (g) KPCAHIST; and (h) A-KPCA; (i) M-KPCA. The classification was done by SVM with an RBF kernel.
(a)
Labeled scene (b)
Raw (c)
PCA (d)
KPCARBF (e)
KPCALap (f)
KPCACau (g)
KPCAHIST (h)
A-KPCA (i)
M-KPCAIn the last experiment, we increase the number of KPCAs in both A-KPCA and M-KPCA utilizing different scale parameters in the same kernel. Tables3 and 4 show that the best single kernel for each data set is different. Therefore, we, respectively, adopt the seven RBF and Cauchy kernel functions for Indian Pines and KSC such as their scale parameters in the range σ0/8,8σ0. The central parameter σ0 is determined by (15). The SVM is again employed for classification after FE. The selection of eigenvalues for each method is defined in 99% confidence interval. Table 5 summarizes the classification accuracies of this experiment. The results show that the M-KPCA-based features are better than the individual KPCAs and A-KPCA features on all data sets, no matter which kernel parameter is applied.Table 5
Overall accuracy, OA (%), and Kappa statistic,κ, of SVMs with different features. The best scores for each data set are highlighted in bold face font, and the second best is underlined.
Data Kernel Metric KPCA A-KPCA M-KPCA σ 0 / 8 σ 0 / 4 σ 0 / 2 σ 0 2 σ 0 4 σ 0 8 σ 0 Indian Pines RBF OA 83.54 88.69 90.52 92.69 57.80 52.41 19.47 94.09 96.03 κ 0.8006 0.8703 0.8918 0.9202 0.5036 0.4405 0.1267 0.9327 0.9459  KSC Cauchy OA 91.83 93.78 93.24 94.34 80.27 72.54 69.76 95.93 96.36 κ 0.9089 0.9307 0.9247 0.9409 0.7794 0.6918 0.6605 0.9547 0.9574


### 4.3. M-KPCA versus Other Dimension Reduction Algorithms

In this section, we compare our method (M-KPCA) with the five FE methods, i.e., LDA, LPP, pPCA, RP, and t-SNE. M-KPCA is constructed with the subkernels indicated in Table2, and kernel parameters are determined from (15). Different values of the dimensionality number of the new subspaces are tested for the SVM classifier across the two data sets. A set of values 1,2,5,10,20,50 are independently generated for the subspace dimension. The classification accuracy is reported for each model, and we plot the results in Figure 6.Figure 6
Classification results of SVM on the two data sets for varying subspace dimension produced by six FE methods.
(a)
Indian Pine (b)
KSCInspection of Figure6 reveals that proposed method regularly outperforms the competing FE methods for multiclass classification in higher dimensions. For instance, if the number of extracted features is set to 50, M-KPCA improves over the best competing method RP by 5.19% in terms of OA on Indian Pines and by 3.78% on KSC. It can be also found from Figure 6 that t-SNE method is highly stable against any dimensional changes. On comparison of methods, we also observe that the performance of LDA and pPCA is limited for both data sets. Considering the lower dimensions (i.e., when the number of new dimension is assigned a value smaller than 10), the best features are produced by the t-SNE which is also the most time-consuming method. The rest of the methods are sorted as RP, LDA, LPP, pPCA, and M-KPCA in ascending order according to the average computation times.


## 4.1. Data Sets and Experimental Setup

The first set is an airborne remote sensing data captured by the AVIRIS sensor over northwest Indiana on June 12, 1992. Indian Pines data has 16 labeled classes and 145 lines/scene and 145 pixels/line. Originally, the scene has 220 spectral bands (10 nm spectral bandwidth from 0.4 to 2.5μm); after discarding the water absorption and noise bands, based on [44, 45], only 159 bands were used in the experiments.Airborne hyperspectral data is acquired by AVIRIS sensor at 18 m spatial resolution over Kennedy Space Center (KSC) during March 1996 and has been employed as a second data source. Noisy bands and water absorption bands are removed. The remaining of the HSI data has 176 bands for 13 wetland and upland classes. Figure1 shows the RGB image of Indian Pines and false color image of the KSC. Table 1 lists the summary of the data sets in our experiments. All samples in each data are adjusted in the range [0,1], as suggested in [46].Table 1
Information for Indian Pines and KSC data sets.
No Indian Pines Samples KSC Samples Class name Train Test Class name Train Test 1 Alfalfa 14 40 Scrub 190 571 2 Corn-no till 359 1075 Willow swamp 61 182 3 Corn–min till 209 625 Cabbage hamm 64 192 4 Corn 59 175 Cabbage palm 63 189 5 Grass-pasture 124 373 Slash pine 40 121 6 Grass-trees 187 560 Oak 57 172 7 Grass-past. moved 6 20 Hardwood swamp 264 791 8 Hay-windrowed 122 367 Graminoid marsh 108 323 9 Oats 5 15 Spartina marsh 130 390 10 Soybean-no till 242 726 Cattail marsh 101 303 11 Soybean-min till 617 1851 Salt marsh 105 314 12 Soybean-clean till 154 460 Mud flats 126 377 13 Wheat 53 159 Water 232 695 14 Woods 324 970 — 15 Bldg-Grass-Tree 95 285 — 16 Stone-steel towers 24 71 —  Total  2594 7772  1541 3670Figure 1
AVIRIS data sets: (a) RGB image of Indian Pines obtained from default bands{R:24,G:14,B:8} and (b) false color composite image of KSC obtained from bands {R:40,G:29,B:20}.
(a) (b)The PCA and KPCAs are implemented using the SIMFEAT toolbox [47]. In each experiment, a single kernel is selected for the KPCA. In addition to Gaussian radial basis function (RBF) kernel which is formulated as K(x,y)=exp⁡(-x-y2/2σ2), we have employed three more kernels (see Table 2). Before solving the eigenvalue problem, the parameter σ∈R+ in the RBF, Laplacian and Cauchy kernels should be selected or optimized. Unless otherwise stated the kernel parameter is set to(15)σ=∑Nxk-x-2N,where x- is the centroid of the total N training data [48]. The kernel parameter σ is nonoptimized and same for each exploited kernel given in Table 2. However, the aim of combination of nonoptimized kernels is to yield a better FE technique for classification.Table 2
Employed kernel functions.
Kernels Formula RBF K R B F ( x , y ) = exp ⁡ - x - y 2 2 σ 2  Laplacian K L a p x , y = exp ⁡ - x - y σ  Cauchy K C a u ( x , y ) = 1 1 + x - y 2 / σ 2  Histogram Intersection (HIST) K H I S T x , y =  ∑ i = 1 N min ⁡ x i , y iThe valuable parts of the obtained cumulative eigenvalues after eigen-decomposition for each method are shown in Figure2. According to the cumulative eigenvalues of PCA, two principal components reach 99% of total variance for Indian Pines case. Nevertheless, in KSC case, three principal components are needed to reach 99% of information. According to these results, the new dimensions of Indian Pines and KSC for classification experiment are, respectively, defined as 2 and 3. However, hyperspectral information cannot be represented utilizing only the second-order statistics as it is pointed out in Section 1. From Figure 2, it can be derived that more kernel principal components (KPCs) are needed to realize the same amount of variance as for PCA. Note that the total number of components with PCA is equal to the number of bands, i.e., 159 for Indian Pines, while it is equal for KPCA to the size of the number of training samples, i.e., 2594, which is significantly higher. For the Indian Pines data set, the first 11, 51, 31, and 33 KPCs are needed to accomplish 99% of the cumulative variance with RBF, Laplacian, Cauchy, and histogram intersection (HIST) kernels, respectively. We observe that 8 KPCs are needed with the RBF, 56 with the Laplacian, 18 with the Cauchy, and 55 with the HIST kernel to achieve same amount of information considering to KSC results. In the case of A-KPCA and M-KPCA,p is set to 1 for kernel selection. For the Indian Pines data set, 35 adaptive KPCs and 20 multiple KPCs contain 99% of information and only 14 adaptive KPCs and 12 multiple KPCs for the KSC.Figure 2
Percentages of cumulative variances of the initial data retained in the components of the new representation for the two data sets: (left to right) first row: PCA andKPCARBF; second row: KPCALap and KPCACau; third row: KPCAHIST and A-KPCA; bottom: M-KPCA.In order to demonstrate the first principal components (PCs) more efficiently, a subimage of size 100 × 100 in the KSC hyperspectral cube is selected. The first PCs for all of the methods are depicted in Figure3.Figure 3
(a) False color image of KSC subscene and most informative principal components (PCs) for the subimage of KSC data: (b) 1st PC; (c) 1st KPCRBF; (d) 1st KPCLap; (e) 1st KPCCau; (f) 1st KPCHIST; (g) 1st A-KPC; (h) 1st M-KPC.
(a)
Original (b)
1st PC (c)
1st KPCRBF (d)
1st KPCLap (e)
1st KPCCau (f)
1st KPCHIST (g)
1st A-KPC (h)
1st M-KPCAfter FE, SVM classifier has been employed for classification. For nonlinear SVMs, we have used the RBF kernel which is formulated in Table2. The classification experiments and the optimization of parameters, C and σ, of SVMs are achieved using LIBSVM [49] with 5-fold cross validation technique. Since SVMs are designed to solve binary problems, various approaches have been proposed for multiclass situations such as remote sensing applications. The most popular approaches for multiclass classification are one-against-all (1AA) and one-against-one (1A1). In this paper, we have applied the 1AA strategy for each class. Each test sample is finally labeled as the class whose output score is maximum.Finally, we compare the proposed M-KPCA algorithm against five state-of-the-art dimension reduction algorithms, i.e., linear discriminant analysis (LDA) [50], LPP, probabilistic PCA (pPCA) [51], RP, and t-SNE. LDA, LPP, pPCA, and t-SNE are implemented using the MATLAB toolbox [52] for dimensionality reduction, and RP algorithm is designed based on Wang’s work [53].


## 4.2. Comparison with KPCA and A-KPCA

The original data sets, termed as raw, are also classified for comparisons. Tables3 and 4 compare the performance of all models numerically (class accuracies and overall accuracy (OA) in percentages) and statistically (kappa test) for the Indian Pines and the KSC data sets, respectively.Table 3
Overall accuracy, OA (%), and Kappa statistic,κ, of the Indian Pines data set with different models. The best scores for each class are highlighted in bold face font, and the second best is underlined.
Feature Raw PCA KPCARBF KPCALap KPCACau KPCAHIST A-KPCA M-KPCA  # of features 159 2 11 51 31 33 35 20  SVM C 430.5 279.17 534.67 663.98 663.98 663.98 603.46 346.67 params σ 2.18 15.22 13.37 13.28 13.27 13.28 16.59 4.74  1 87.04 33.34 88.89 74.07 77.78 90.74 90.74 94.44 2 88.77 40.59 86.82 75.80 82.08 83.19 89.82 89.96 3 86.09 06.95 86.81 65.47 82.61 78.30 88.49 88.60 4 92.31 35.90 91.03 68.38 81.62 82.48 91.45 96.58 5 98.19 58.95 97.79 88.73 95.77 90.74 97.59 98.59 6 98.93 91.43 98.39 97.59 97.99 96.52 99.06 99.33 7 92.31 30.77 92.31 88.46 92.31 84.62 92.31 96.15 8 99.39 95.50 98.98 97.75 98.98 97.34 99.39 99.59 9 90.00 00.00 95.00 90.00 85.00 90.00 95.00 95.00 10 89.67 66.32 91.12 80.58 90.50 88.74 91.22 91.84 11 91.90 73.30 93.27 86.95 91.13 89.55 93.11 94.25 12 94.30 21.01 95.11 80.78 90.88 88.27 95.44 95.60 13 99.06 73.58 99.06 98.11 98.58 97.64 99.06 99.53 14 97.30 86.79 98.07 96.91 97.99 96.52 97.84 97.60 15 80.00 15.53 77.11 53.42 70.26 58.95 82.63 88.95 16 92.63 90.53 94.74 94.74 91.58 90.53 90.53 94.74  OA 92.47 59.78 92.69 83.92 90.16 88.30 93.44 94.28 κ 0.9130 0.5399 0.9210 0.8215 0.8967 0.8749 0.9317 0.9348Table 4
Overall accuracy, OA (%), and Kappa statistic,κ, of the KSC data set with different models. The best scores for each class are highlighted in bold face font, and the second best is underlined.
Feature Raw PCA KPCARBF KPCALap KPCACau KPCAHIST A-KPCA M-KPCA  # of features 176 3 8 56 18 55 14 12  SVM C 534.66 279.17 430.54 663.98 181.02 346.69 117.38 165.69 params σ 3.77 9.87 34.95 5.58 10.69 5.58 5.61 5.01  1 97.50 92.64 97.50 95.66 96.58 96.32 98.03 98.29 2 94.24 84.77 93.83 90.12 91.77 91.36 94.24 95.47 3 93.75 89.84 93.36 91.80 93.75 93.75 93.36 97.26 4 78.17 49.21 80.16 71.43 78.17 75.40 81.35 93.25 5 77.64 56.52 60.87 55.28 61.49 57.76 78.88 88.82 6 75.11 50.66 58.08 55.02 75.11 58.95 82.53 86.90 7 93.34 73.33 89.52 91.43 90.48 91.43 94.29 95.24 8 96.29 78.65 94.20 90.26 95.13 91.18 96.75 96.28 9 99.04 97.50 98.65 96.73 98.65 97.50 99.23 99.61 10 100 91.58 99.75 97.77 99.26 97.77 100 99.26 11 99.05 97.85 99.05 98.09 99.05 98.81 99.05 99.28 12 99.06 81.11 96.82 95.23 97.42 95.63 99.01 99.20 13 100 99.89 100 99.89 99.89 99.89 99.89 100  OA 95.51 86.53 93.78 91.65 94.34 92.59 96.14 97.52 κ 0.9522 0.8493 0.9347 0.9106 0.9407 0.9213 0.9601 0.9724Inspection of Table3 reveals A-KPCA outperforms PCA and all the four KPCAs. Further analysis shows that the KPCA performs significantly better than the conventional PCA. Regarding the OAs, it is clear that the M-KPCA based classification produces more accurate results when compared to the A-KPCA based classification. RBF kernel gives the best results for KPCA among the other kernel functions as seen in Table 3.The results for the KSC data set are reported in Table4. Regarding the PCA and KPCA results, FE does not improve the accuracies significantly. The comparison between KPCA and PCA shows that KPCA performs better than the PCA in terms of classification accuracies. Moreover, classification of the A-KPCA features is more precise that the one yielded employing the all KPCs. As with the previous experiment, the best results are obtained with the M-KPCA. Figures 4 and 5 represent the available labeled scenes and classification maps of all models for the Indian Pines and KSC data sets, respectively.Figure 4
Classification results obtained with the Indian Pines: (a) labeled scene and classification maps using the (b) raw data; (c) PCA; (d) KPCARBF; (e) KPCALap; (f) KPCACau; (g) KPCAHIST; (h) A-KPCA; (i) M-KPCA. The classification was done by SVM with an RBF kernel.
(a)
Labeled scene (b)
Raw (c)
PCA (d)
KPCARBF (e)
KPCALap (f)
KPCACau (g)
KPCAHIST (h)
A-KPCA (i)
M-KPCAFigure 5
Classification results obtained with the KSC: (a) labeled scene and classification maps using the (b) raw data; (c) PCA; (d) KPCARBF; (e) KPCALap; (f) KPCACau; (g) KPCAHIST; and (h) A-KPCA; (i) M-KPCA. The classification was done by SVM with an RBF kernel.
(a)
Labeled scene (b)
Raw (c)
PCA (d)
KPCARBF (e)
KPCALap (f)
KPCACau (g)
KPCAHIST (h)
A-KPCA (i)
M-KPCAIn the last experiment, we increase the number of KPCAs in both A-KPCA and M-KPCA utilizing different scale parameters in the same kernel. Tables3 and 4 show that the best single kernel for each data set is different. Therefore, we, respectively, adopt the seven RBF and Cauchy kernel functions for Indian Pines and KSC such as their scale parameters in the range σ0/8,8σ0. The central parameter σ0 is determined by (15). The SVM is again employed for classification after FE. The selection of eigenvalues for each method is defined in 99% confidence interval. Table 5 summarizes the classification accuracies of this experiment. The results show that the M-KPCA-based features are better than the individual KPCAs and A-KPCA features on all data sets, no matter which kernel parameter is applied.Table 5
Overall accuracy, OA (%), and Kappa statistic,κ, of SVMs with different features. The best scores for each data set are highlighted in bold face font, and the second best is underlined.
Data Kernel Metric KPCA A-KPCA M-KPCA σ 0 / 8 σ 0 / 4 σ 0 / 2 σ 0 2 σ 0 4 σ 0 8 σ 0 Indian Pines RBF OA 83.54 88.69 90.52 92.69 57.80 52.41 19.47 94.09 96.03 κ 0.8006 0.8703 0.8918 0.9202 0.5036 0.4405 0.1267 0.9327 0.9459  KSC Cauchy OA 91.83 93.78 93.24 94.34 80.27 72.54 69.76 95.93 96.36 κ 0.9089 0.9307 0.9247 0.9409 0.7794 0.6918 0.6605 0.9547 0.9574


## 4.3. M-KPCA versus Other Dimension Reduction Algorithms

In this section, we compare our method (M-KPCA) with the five FE methods, i.e., LDA, LPP, pPCA, RP, and t-SNE. M-KPCA is constructed with the subkernels indicated in Table2, and kernel parameters are determined from (15). Different values of the dimensionality number of the new subspaces are tested for the SVM classifier across the two data sets. A set of values 1,2,5,10,20,50 are independently generated for the subspace dimension. The classification accuracy is reported for each model, and we plot the results in Figure 6.Figure 6
Classification results of SVM on the two data sets for varying subspace dimension produced by six FE methods.
(a)
Indian Pine (b)
KSCInspection of Figure6 reveals that proposed method regularly outperforms the competing FE methods for multiclass classification in higher dimensions. For instance, if the number of extracted features is set to 50, M-KPCA improves over the best competing method RP by 5.19% in terms of OA on Indian Pines and by 3.78% on KSC. It can be also found from Figure 6 that t-SNE method is highly stable against any dimensional changes. On comparison of methods, we also observe that the performance of LDA and pPCA is limited for both data sets. Considering the lower dimensions (i.e., when the number of new dimension is assigned a value smaller than 10), the best features are produced by the t-SNE which is also the most time-consuming method. The rest of the methods are sorted as RP, LDA, LPP, pPCA, and M-KPCA in ascending order according to the average computation times.


## 5. Conclusion

In this paper, a novel semisupervised KPCA framework named multiple KPCA (M-KPCA) is proposed for effective feature extraction of hyperspectral images. It applies ensemble strategy to favor good candidate kernels during nonlinear projections. A noniterative algorithm is developed to simultaneously feature extraction and kernel combination based on a kernel class separability criteria. In terms of the number of kernels, KPCA uses only one base kernel with predefined parameter(s) (if existing). In terms of the kernel quality, A-KPCA has no measurement procedure to evaluate the efficiency of kernels. M-KPCA overcomes these drawbacks of both KPCA and A-KPCA.Dimension reduced HSI data is classified by nonlinear SVMs to compare classification performance for several models. Experiments on two real HSI data sets demonstrate that the best kernel type varies according to data (see Tables3 and 4). In the first test, KPCA presents better performance compared to the conventional PCA. Overall evaluation for dimension reduction performance of PCA, KPCAs, A-KPCA, and M-KPCA techniques shows that the M-KPCA is more successful than the others. In the second experiment, we have employed seven candidate kernel functions using different kernel parameters for each data. These KPCAs are then utilized to construct the A-KPCA and M-KPCA. Experiments on the AVIRIS data sets confirm that the M-KPCA outperforms the individual KPCAs and the A-KPCA in terms of both OA and Kappa coefficient. Moreover, the comparative results in Section 4.3 demonstrate that M-KPCA experimentally accomplished superior or competitive classification accuracy more than the other unsupervised state-of-the-art FE methods.The results clearly validate that semisupervised learning of kernels with M-KPCA increases the robustness of nonoptimized KPCAs. One and probably most important limitation of the M-KPCA is its computational complexity, related to the number of samples used for constructing the kernel matrix. Therefore, our future work aims to address the problem of reducing the complexity. It is also possible to extend the proposed method to a selective approach which eliminates weak kernels before feature extraction.



---

*Source: 9632569-2018-09-06.xml*