計測自動制御学会東北支部 第 340 回研究集会 (2022.12.23)
資料番号 340-1

カーネル法による線形主成分判別分析
Kernelized Linear Principal Component Discriminant Analysis
曲凌曉 ∗ ，裴岩 ∗∗○ Qu Lingxiao∗ , Pei Yan∗∗*会津大学 大学院コンピュータ理工学研究科，**会津大学 コンピュータ・サイエンス部門*Graduate School of Computer Science and Engineering, University of Aizu,**Division of Computer Science, University of Aizu
キーワード : 主成分分析 (Principal Component Analysis)，線形判別式分析 (Linear Discriminant
Analysis)，カーネル法 (Kernel method)

連絡先 : 〒 965-8580 日本国福島県会津若松市一箕町鶴賀 会津大学 裴岩，Tel.: +81-(242)-37-2765

1.

E-mail: peiyan@u-aizu.ac.jp

Introduction

Principal Component Analysis (PCA) 26) andLinear Discriminant Analysis (LDA) 8) are twostatistical learning algorithms using coordinatetransformation for dimension reduction of highdimensional data.

They are applied widelyin feature extraction, data classiﬁcation, andclustering problems.
PCA is an unsupervised learning method,
pursuing some projection directions with the
maximum sum of the total variance of projected data.

It is useful for feature mining

and extraction by reducing higher dimensional
data to lower dimensional data 1) . LDA isa supervised learning method with the objective that ﬁnding the projection direction withthe maximum distance of projections of classprojections of sample data in each class and the
projections of corresponding class mean 12) .
LDA has good performance in data classiﬁcation and clustering problems.
PCA and LDA are used to solve linear problems in low-dimensional space. Kernel methods 10) can be used to increase data dimension, and transform the nonlinear models in
low dimensional space to linear ones in high dimensional space. It simpliﬁes the complexityof problems using the transformation. Thereis research on kernel PCA 21) , referred to as
KPCA in the following, and Generalized Discriminant Analysis (GDA) 3) . These researches
perform PCA and LDA in high dimensional
space by using kernel methods to handle nonlinear problems 17) .

means and the minimum distance between the–1–

KPCA has many extensions of its techniques,sum of distances between projected classalgorithms, and applications. For example, sparse
kernel PCA 23) , robust kernel PCA 18) , incre-means in feature space;mental kernel PCA 4) , adaptive kernel PCA 6) ,3) pursuing projection vectors of minimum

and streaming kernel PCA 9) , etc. KPCA has

sum of distances between projected classbeen applied in kinds of scenes, including facerecognition 14) , image modling 13) , fault detac-

data in each class and the corresponding
projected class mean in feature space.

tion 5) , and geostatistics 20) , etc. There arealso many pieces of research on GDA, which

It is referred to as Kernelized Linear Principal

apply GDA with diﬀerent approaches and ex-

Component Discriminant Analysis (KLPCDA).

tend GDA to multiple situations. For instance,
GDA based on distance 2) , GDA of matrix

The implementation of kernel extension in LPC-

exponential approach 29) , modiﬁed GDA preventing eigenvalue degenerating 30) , as well as

ysis methods in high dimensional space is the

GDA with generalized singular value composition 11) , etc. In addition, the applications of

The following of this paper is organized as

DA and the presence of varieties of data analoriginality of this work.
below. We illustrate the theoretical implemen-

GDA to tree-structured classiﬁcation 15) , feature extraction with DNN 22) , face recogni-

tation of our proposed KLPCDA in the sec-tion 16) , and under-sampled problems 27, 28) ,Discriminant Analysis”. We show our exper-

etc., have been hot topics for a long time.

iment and evaluation in the section ”Evalua-tion ”Kernelized Linear Principal Component

Pei proposed a framework of data analysistion”. The analysis and discussion of evalu-

methods called Linear Principal Component

ation results and the problems that remained

Discriminant Analysis (LPCDA) using the samecharacteristics of PCA and LDA 19) . It has

are presented in the next section. Finally, we

the advantages of both supervised and unsu-

tential study subjects.

conclude the whole work and present the po-

pervised learning methods. In this paper, we
extend LPCDA to solve nonlinear problems inhigh dimensional space using kernel methods.

2.

It can be used for both classiﬁcation and clustering problems, so we refer to it as a series

Kernelized Linear Principal
Component Discriminant Analysisof semi-supervised learning methods. We pro-

A series of data analysis methods called Lin-

pose a framework of data analysis methods for

ear Principal Component Discriminant Anal-

nonlinear problems in high dimensional spaceysis, which is a uniform framework based onby combining three objectives of KPCA and

Principal Component Analysis and Linear Discriminant Analysis 19) , was proposed, as shown

GDA as follows,
1) pursuing projection vectors of the maxi-in Table 1. In this work, we establish the kernel

mum total variance of projected data in

implementation of proposed methods in Linear

feature space;

Principal Component Discriminant Analysis,
by using the kernel trick. It pursues a pro-2) pursuing projection vectors of maximum

jected direction v in feature space H and sat-–2–

isﬁes: (a) with the maximum total variance of
projected samples 21) ; (b) with the maximum
sum of the distance of the projected group’s
center points 3) ; and (c) with minimum sum of
the inner variance of the projected groups 3) .
Given n training samples of column vectors
x1 , x2 , · · · , xn ∈ Rd in original space.
L
P
Given n =ni training samples f column
i=1(1)

Table 1
Methods of principal component
discriminant analysis. There are three objectives in LPCDA 19) , i.e., arg max v T Cv,
arg max v T Sb v, and arg min v T Sw v. With the
combinations of these three objectives, there
are seven methods in LPCDA, making up the
proposed uniform framework, containing the
PCA and LDA. This table is adopted from reference 19) .
Method No.

(1)vectors with labels 1, · · · , L: x1 , · · · , xn1 , · · · ,
(i)

(i)

(L)

xj , · · · , xnL in original space, where xj
Rd , i ∈ [1, L],

1
2
3
4
5
6
7

∈

j ∈ [1, ni ] is the j-th sample

in class i. ni and L are the number of i-thclass and total labels, respectively. They aremapped into a feature space and referred respectively as X T = ϕ(x1 ), ϕ(x2 ), · · · , ϕ(xn ) ,

(1)
(1)
(i)
and X T = ϕ(x1 ), · · · , ϕ(xn1 ), · · · , ϕ(xj ), · · · ,(L) ϕ(xnL ) .

target function

meaningv T Cv+v T Sb v
v T Sw v
v T Cv + v T Sb v
v T Sb v
v T Sw v
v T Cv

Sb v
LPCDA: v Cv+v
v T Sw v
T
LPCDA: v Cv + v T Sb v
linear discriminant analysis
principal component analysis
T
LPCDA: vvT SCv
wv
T
LPCDA: v Sb v
LPCDA: v T Sw v

Tv T Cv
v T Sw v
v T Sb v
v T Sw v

Sb =

L
X
ni
i=1

n

T

(mi − m)(mi − m)T .

(3)

The objective (a) is presented by Eq. (1).
The objective (b) is presented by Eq. (2) andv = arg(3), and (c) is presented by Eq. (4) and (5).
In these equations, mi , mj and m represent the= argmeans of the projections of class i, j and all

ni
L
X
ni X

min

v∈Rd ,|v|=1

minv∈Rd ,|v|=1

i=1
T

n

(i)

(v T ϕ(xl ) − mi )2

l=1

v Sw v.
(4)

samples, respectively. mi and m represent themeans of class i and all samples. These objectives are all implemented in the feature space.
Baudat etc. didn’t consider each group’s center in objective (c) 3) . We consider it and pur-

Sw =

ni
L
X
ni X
i=1

n

(i)

(i)

(ϕ(xl ) − mi )(ϕ(xl ) − mi )T .

l=1

(5)

sue the distance between projections of each

We will introduce the theoretical implemen-group’s samples and this group’s center is thetation of kernelizing ﬁve out of seven methods

minimum, as shown in Eq.s (4) and (5).

in Table 1, where No.3 and No.4 methods havebeen kernelized before, that is kernel PCA and
GDA, respectively. We weight the objectives

v = argv = arg= arg

maxv∈Rd ,||v||=1max

σ 2 = arg

1

L X
L
X

v∈Rd ,||v||=1 2

maxv∈Rd ,||v||=1

i=1 j=1

v T Cv.

(a), (b), and (c) by parameter α, β, and γ, that(1)is, αv T Cv, βv T Sb v, γv T Sw v, respectively. In

ni nj(mi − mj )2n n

the following deduction, we will transform the

maxv∈Rd ,||v||=1

combinations of three objectives to eigenvalue

T

v Sb v.

problems containing matrices C, Sb , Sw by La(2)–3–grange multiplier. However, in feature space,C=

can be deduced as follows,

1 T
X X, Sb = X T BX, Sw = X T W X,nf (v, λ) = v T Cv + v T Sb v − λ(v T Sw v − 1),∂f= 2Cv + 2Sb v − 2λSw v = 0,
(8)
∂v
∂f
= v T Sw v − 1 = 0.
∂λ

(6)
are unknown, where1111
B = diag( 1n1 ×n1 , · · · ,1n ×n ) − 2 1n×n ,nn1
nL L Ln
W = diag(P1 In1 , · · · , PL InL )11− diag( 1n1 ×n1 , · · · , 1nL ×nL ),nn
can be known 3, 21) .

From Eq. (8), we obtain(C + Sb )v = λSw v,(9)which is a generalized eigenvalue problem. The
objective can be deduced as

1ni ×ni , i ∈ [1, L] are ani × ni matrix with all entries 1. In order tov = arg

solve the eigenvalue problems, we have to apply Wanba theory 25)

= arg

max

v T Cv + v T Sb v

max

λ.

||v T Sw v||=1||v T Sw v||=1

Therefore, the optimal v corresponds to thev = X T α,

(7)maximum eigenvalue λ of Eq. (9). From Eq.

and kernel matrix K = XX T to the deduction 10) . Schölkopf etc. 21) implemented cen-(6), and Wahba Theory Eq. (7), with kernel

tering in high-dimensional space by computing
e from K,
centered kernel matrix K

matrix K = XX T , the eigenvalue problem (9)can be dealt as below.
1( X T X + X T BX)v = λX T W Xv,
ne = K− 1 1n×n K− 1 K1n×n − 1 1n×n K1n×n ,
K
n
n
n2

1
XX T Xv + XX T BXV = λXX T W Xv,
nwhere 1n×n are a n×n matrix with all entries 1.
e to operate
We replace K with the centered K

1
XX T XX T α + XX T BXX T α = λXX T W XX T α,n1(W K)−1 ( K + BK)α = λα. (10)nthe centering in practice.

In Eq. (10), matrices K, W, B, and parame-

2.1

ter n are all known. Hence we can solve this

Method No.1

eigenvalue problem and ﬁnd eigenvalues λ and

T
TS v
b
The target function is v = arg max αv Cv+βv. eigenvectors α. We suppose ||v|| = 1 for target
γv T Sw v

To simplify the calculation, we set α, β, andv = X T α, so it has to be normalized as

γ = 1 in the following. We can solve the problem by the Lagrange multiplier. About the objective (c), we set it as a constraint ||v T Sw v|| =1 in the following calculation. The objective ofMethod No.1 can be presented by v = arg max||vT Sw v||=1
v T Cv + v T Sb v. From Lagrange multiplier, it

–4–

v=

XT α
XT α
√
=
.
||X T α||αT KαSuppose there are new data x′ , its projections

the authors didn’t handle the centering in fea-

onto v in a feature space areture space and didn’t take the distance of pro

jections between class samples and the corre-k(x1 , x′ )


..


.




(1)
′
 k(xn1 , x ) 


1


..
v T ϕ(x′ ) = √
αT 
.
.
T


α Kα
k(x(L) , x′ )


1


..


.


(L)
k(xnL , x′ )

2.2

(1)

sponding class means into consideration, which
is dealt with by us in the following.
T

βv Sb v
The target function is v = arg max γv
TS v.
w

As mentioned in Eq. (5), it contains the classmeans. The objective can be presented as v =arg max||vT Sw v||=1 v T Sb v. By Lagrange multiplier,Sb v = λSw v.

Method No.2

(13)

The target function of this method is v =With kernel matrix, Eq. (6), and Eq. (7),arg max αv T Cv +βv T Sb v. With Lagrange mul-

we calculate Eq. (13) and obtain the solvabletiplier, it’s objective can be presented as v =eigenvalue problem Eq. (14).

arg maxv∈Rd ,||v||=1 v T Cv + v T Sb v, and can be

X T BXv = λX T W Xv,

calculated to obtain this eigenvalue problem,(C + Sb )v = λv.

(W K)−1 BKα = λα.

(11)

2.4

The objective can be simpliﬁed asv = arg
= arg

(14)

Method No.4

max

v T Cv + v T Sb v

Method No.4 is kernel PCA and it was implemented in 1996 21) . The target function

max

λ.

is v = arg max αv T Cv. By Lagrange multi-v∈Rd ,||v||=1
v∈Rd ,||v||=1

plier, the objective can be presented as v =The target v corresponds to the largest λ ofarg maxv∈Rd ,||v||=1 v T Cv, and be transformed

Eq. (11). By applying kernel matrix, Eq. (6),to the eigenvalue equation Eq. (15)and Eq. (7), we can transform Eq. (11) and1 T
X Xv = λv.
nsolve the eigenvalue problem Eq. (12).
1( X T X + X T BX)v = λv,
n1( K + BK)α = λα.
nFrom Eq. (6) and (7), and kernel matrix, Eq.
(15) can be deduced a solvable one Eq. (16).
(12)

1Kα = λα.
n

2.5
2.3

(15)

Method No.3

(16)

Method No.5

For this method, the target function is v =TMethod NO.3 is GDA and its implementation in feature space using the kernel approachhas been ﬁnished in the year 2000 3) . While

αv Cv
arg max γv
T S v . From Lagrange multiplier, thewobjective can be denoted as v = arg max||vT Sw v||=1
v T Cv. By further calculation, we obtain
Cv = λSw v.

–5–

(17)

original data

4.5

0.16
1
2
3

4

KLPCDA Method No.1

0.8
1
2
3

0.15

KLPCDA Method No.2

Method No.3 GDA

1
1
2
3

0.6

1
2
3

0.5

0.4
0.14
3.5

Y

Y

Y

Y

0.2
0.13

0

0

3
0.12

-0.2
2.5

0.11

2
4

5

6

7

0.1
0.1

8

0.12

X

0.14

0.16

0.18

-0.6
-1

0.2

-0.5

X

Method No.4 KPCA

0.8

-0.5

-0.4

0.1
1
2
3

0.6

0

0.5

-1
-0.2

1

-0.1

X

KLPCDA Method No.5

0.3
1
2
3

0.05

KLPCDA Method No.6

0.1

0.2

KLPCDA Method No.7

0.15
1
2
3

0.2

0

0

X

1
2
3

0.1

0.1

0.4

Y

Y

0

Y

Y

-0.05
0.2

-0.1

-0.1

-0.15

-0.2

-0.2

-0.3

0.05

0

-0.2

-0.4
-1

-0.5

0

0.5

1

-0.25
-1

-0.5

X

0

0.5

-0.4
-1

1

0

-0.05
-0.5

X

0

0.5

1

-0.2

-0.1

X

0

0.1

0.2

X

Fig. 1 Projections of seven kernelized data analysis methods on Fisher Iris data set by using
Gaussian kernel with σ = 0.2. The labels 1, 2, 3 correspond to the classes Iris-setosa, Iris-versicolor,and Iris-virginica, respectively. The X axis is the ﬁrst principal component, and the Y axis is thesecond principal component.
With Eq. (6), kernel matrix, and Wahba Rep-

the application of Eq. (6) and (7), and kernel

resenter Theory, we deduce Eq. (17) and ob-

matrix.

tain the eigenvalue equation (18).
1 T
X Xv = λX T W Xv,n1(W K)−1 Kα = λα.
n

X T BXX T α = λX T α,
BKα = λα.
(18)

(20)

We can solve the eigenvalue problem Eq. (20).

We can work out α and λ.

2.7
2.6

Method No.6

Method No.7

The target function of this method is v =The target function of this method is v =arg min v T Sw v. By Lagrange multiplier, the ob-

arg max βv T Sb v. With the Lagrange multiplier,jective can be represented as v = arg minv∈R,||v||=1

we can present and deduce our objective as

v T Sw v, and be deduced tov = arg maxv∈Rd ,||v||=1 v T Sb v, and the eigen-Sw v = λv.

value equation
Sb v = λv.

(19)

(21)

With the usage of Wahba Representer Theory,
the kernel matrix, and Eq. (6), we deduce theWe transform the above eigenvalue equation
Eq. (19) to the following solvable one with

–6–

Table 2 Total variance of projected samples, each projected inner-class variance, and sum ofsquare-distances between projected class-means conducted on Fisher Iris data set of Gaussian
kernel, σ = 0.2, for each method. The bold data shows the optimal results of same items among
seven methods.
Method No.
1
2
3
4
5
6
7

Total variance6.2901e-04
0.08140.01670.08300.01750.05740.0216

Total variance of class 1
7.6961e-11
0.1562
1.7943e-11
0.17492.2640e-13
0.02354.4864e − 31

Total variance of class 2
1.8332e-100.00164.2740e-11
0.001279
5.3928e-13
0.00751.4725e − 31

Total variance of class 39.2548e-09
4.4162e-04
2.1577e-09
3.4076e-04
2.7225e-11
0.00302.0750e − 31

distance between two classes
0.0056
0.26240.1496
0.22270.15630.41290.1927

Table 3 Total variance of projected samples, each projected inner-class variance, and sum ofsquare-distances between projected class-means conducted on Fisher Iris data set of Polynomial
kernel, r = 3, for each method. The bold data shows the optimal results of same items among
seven methods.
Method No.
1
2
3
4
5
6
7

Total variance1.6533e-04
1.0052e+05
1.2196e-08
1.0060e+05
9.2758e+04
1.0142e + 05
7.5838e-25

Total variance of class 1
2.2965e-06
714.0925
9.3949e-11778.7099
179.2304
1.4489e+03
1.1093e − 25

Total variance of class 22.8006e-05
1.5101e+04
1.8823e-09
1.5358e+04
1.1506e+04
1.5166e+04
2.2541e − 25

eigenvalue equation (21) and obtain

distance between two classes
0.00117.1050e+05
8.5849e-08
7.0861e+05
6.7552e+05
7.1525e + 05
3.1617e-24

projections of each class mean, we evaluateeach method visually and quantitatively.

X T W Xv = λv,
W Kα = λα.

Total variance of class 3
8.4096e-05
4.8193e+04
5.9085e-09
4.8745e+04
4.0608e+04
4.8512e+04
7.9985e − 25

(22)

3.1

Evaluation on Fisher Iris Data

The eigenvalue problem Eq. (22) is solvable.
In this data set, there are 150 instances with

Now we ﬁnish the theoretical deduction of
our proposed ﬁve methods, kernel PCA andfour attributes: sepal length, sepal width, petal
length, and petal width. It contains 3 classes

GDA.

of 50 instances each: Iris-setosa, Iris-versicolor,and Iris-virginica. In this data set, one class3.

Evaluation

is linearly separable from the other two andIn this section, we evaluate our proposed
KLPCDA using Fisher Iris 7) and wine 24)

the latter are not linearly separable from eachdata sets. We choose two kernel methods, Gaus-

samples are centered in feature space by a cen-sian kernel

tered kernel matrix.

k(x, z) = exp(

other.

In the following evaluations, all the

Figure 1 and Figure 2 show the classiﬁca-−|x − z|2),
2σ 2tion results of projections of the samples onto
the ﬁrst two principal component axes of ﬁveand Polynomial kernel

methods in our proposed KLPCDA, KPCA,

k(x, z) = (< x, z > +1)r .

and GDA on Fisher Iris data set by using Gaus-

By calculating the total variance, the total vari-sian kernel and Polynomial kernel, respectively.

ance of each class, and the distance between

Table 2 and Table 3 present the quantitative–7–

original data

4.5

0.04
1
2
3

4

KLPCDA Method No.1

KLPCDA Method No.2

500
1
2
3

0.03

-1.15
1
2
3

×10 -3 Method No.3 GDA
1
2
3

-1.2
-1.25

0.02

0

3.5

Y

Y

Y

Y

-1.3
0.01

-1.35

3
0

-500
-1.4

2.5

-0.01

2
4

5

6

7

-0.02
-1

8

-1.45

-0.5

X

Method No.4 KPCA

500

0

0.5

-1000
-1

1

-1.5
-0.5

X

400
1
2
3

0

0.5

1

-1.4

X

KLPCDA Method No.5

200

KLPCDA Method No.6

150
1
2
3

-1.2

-1
×10 -3X

KLPCDA Method No.7
164.8008

1
2
3

100

1
2
3

164.800800164.8008
50

Y

Y

Y

Y

-200
-400

0

164.8008
164.8008

-500
-600

164.8008-50-800

164.8008
-1000
-1

-0.5

0

0.5

1

-1000
-1

X

-0.5

0

0.5

-100
-1000

1

X

-500

0

500

1000

X

-1

-0.5

0

0.5

X

Fig. 2 Projections of seven kernelized data analysis methods on Fisher Iris data set by using
Polynomial kernel with r = 3. The labels 1, 2, 3 corresponds to the classes Iris-setosa, Iris-versicolor, and Iris-virginica, respectively. The X axis is the ﬁrst principal component, and the Y axis
is the second principal component.
evaluations of seven methods in KLPCDA withnel, respectively. Table 4 and Table 5 present

Gaussian kernel and Polynomial kernel, respec-

the quantitative evaluation of seven methods

tively.

in KLPCDA with Gaussian kernel and Polynomial kernel, respectively.

3.2

Evaluation on Wine Data

In this dataset, there are 178 wine samples of4.

Analysis and Discussion

three classes labeled 1, 2, and 3. It has thirteen

4.1

Discussion on Results from Fisher
Iris Dataattributes: 1) Alcohol, 2) Malic acid, 3) Ash,
4) Alcalinity of ash, 5) Magnesium, 6) Total
phenols, 7) Flavanoids, 8) Nonﬂavanoid phenols, 9) Proanthocyanins, 10) Color intensity,11) Hue, 12) OD280/OD315 of diluted wines,
and 13) Proline. Class 1, 2, and 3 have 59, 71,
and 48 samples, respectively.
Figure 3 and Figure 4 present the visual evaluation of our proposed ﬁve methods in KLPCDA,
GDA, and KPCA, conducted on Wine data set
by using Gaussian kernel and Polynomial ker-

In Figure 1 of Gaussian kernel on Fisher
Iris dataset, we choose parameter σ = 0.2.
Our proposed methods NO.1, NO.5, NO.6, andNO.7 have perfect classiﬁcation results as GDA,
all of which completely separate three classes.
Method NO.2 has a similar but a little bit
better result than KPCA. Each of our proposed methods can classify data clearly andperform not inferior and even better than GDA–8–1

original data

6

0.1
1
2
3

5

KLPCDA Method No.1

1

KLPCDA Method No.2

1
2
3

0.05

0.2
1
2
3

Method No.3 GDA
1
2
3

0.15

0.5
4

Y

3

Y

0.1

Y

Y

0
0

-0.05

0.05

2
-0.5
-0.1

1

0
10

11

12

13

14

-0.15
-1

15

0

-0.5

X

Method No.4 KPCA

0.8

0

0.5

-1
-0.2

1

-0.1

X

0.15
1
2
3

0.6

0

0.1

-0.05
-0.2

0.2

-0.1

X

KLPCDA Method No.5

0.06
1
2
3

0.1

0

0.1

0.2

X

KLPCDA Method No.6

0.2
1
2
3

0.04
0.02

KLPCDA Method No.7
1
2
3

0.15

0.4
0
0.05

0

0.1

Y

Y

-0.02

Y

Y

0.2

-0.04

0

0.05

-0.06
-0.2
-0.08

-0.05
-0.4

0

-0.1

-0.6
-1

-0.5

0

X

0.5

1

-0.1
-1

-0.5

0

0.5

1

-0.12
-0.2

X

-0.1

0

X

0.1

0.2

-0.05
-1

-0.5

0

0.5

X

Fig. 3 Projections of seven kernelized data analysis methods on Wine data set by using Gaussian
kernel with σ = 0.2. The X axis is the ﬁrst principal component, and the Y axis is the second
principal component.
or KPCA. Table 2 presents the quantitative

as that of KPCA, which have diﬀerent projec-

evaluation results, from which we know KPCA

tion directions from that of GDA. In general,

has the maximum total variance which is tightly

our proposed methods NO.1, NO.2, NO.5, andlarger than that of our proposed method NO.2.

NO.6 have not a bad performance but are infe-

Our proposed method NO.7 has the minimumrior to that of using a Gaussian kernel. Mean-sum of each inner-class variance. Our proposed

while, these methods have similar classiﬁcation

method NO.6 has the maximum sum of squareeﬀects to KPCA and GDA. From Table 3, ourdistances between projected class means. Our

proposed method NO.6 has the maximum total

proposed methods NO.5 and NO.7 have better

variance of projected data and sum of squarevalues on every evaluation metric item than

distances between projected class means, while

GDA, and our proposed methods have bettermethod NO.7 has the minimum sum of eachvalues on some of the evaluation metric items

projected inner-class variance. Consequently,

than KPCA.

it indicates that our proposed method NO.7

Figure 2 and Table 3 show the evaluation re-

has advantages over GDA and KPCA on inner-sults conducted on the Fisher Iris data set withclass variances. And method NO.6 has betterthe polynomial kernel, where we set the param-

values on evaluation metric items of total vari-eter as r = 3. From the ﬁgure, we can visuallyances and the sum of the square distance be-observe that the classiﬁcation results of meth-tween class means than GDA and KPCA.

ods NO.1, NO.2, and NO.5 are nearly the same

–9–

1

Table 4 Total variance of projected samples, each projected inner-class variance, and sum ofsquare-distances between projected class-means conducted on Wine data set of Gaussian kernel,
σ = 0.2, for each method. The bold data shows the optimal results of same items among sevenmethods.
Method No.
1
2
3
4
5
6
7

Total variance0.00570.00560.01460.01130.00570.01130.0069Total variance of class 1
4.5842e-17
2.4168e − 32
6.1464e-16
3.1197e-05
4.3761e-17
2.4011e-31
1.7207e-31

Total variance of class 2
2.47689e − 04
0.00100.00580.02013.2754e-04
0.00473.2759e-04

Total variance of class 3
0.00530.00850.01210.02020.00490.00860.0063

distance between two classes
0.01540.0082
0.06168.2849e-04
0.01640.05080.0190

Table 5 Total variance of projected samples, each projected inner-class variance, and sum ofsquare-distances between projected class-means conducted on Wine data set of Polynomial kernel,
r = 3, for each method. The bold data shows the optimal results of same items among sevenmethods.
Method No.
1
2
3
4
5
6
7

4.2

Total variance1.9676e+14
7.0470e+173.3206e+03
7.0471e+1707.0519e + 17
1.8822e-11

Total variance of class 1
2.6755e+149.5720e+174.5179e+03
9.5738e+1709.5806e+171.3672e − 11

Total variance of class 2
1.1996e+144.3000e+172.0235e+03
4.2994e+1704.3063e+175.9977e − 12

Discussion on Results from Wine
data

Figure 3 shows the results on Wine data
set of Gaussian kernel with parameter σ =0.2. Every of our proposed ﬁve methods aswell as GDA has great classiﬁcation results.
It indicates that our proposed methods perform much better than KPCA. As the quantitative evaluation results in Table 4 show, GDA
has the maximum total variance of projected
samples and the sum of square distances between projected class means, while our proposed methods NO.2, NO.1, and NO.5 havethe minimum sum of inner-class variances ofprojected classes 1, 2, and 3, respectively. In

Total variance of class 35.3108e+121.9086e+16
89.4830
1.9075e+1601.9208e+165.4393e − 12

distance between two classes
8.4263e+14
3.0195e+18
1.4216e+04
3.0193e+1803.0197e + 18
6.8830e-11

From Figure 4 obtained from the Wine data
set of the polynomial kernel with parameterr = 3, all the results are not good. During
our calculation, we met complex numbers anddealt with them by deleting the image partand leaving the real part.

It leads to thatthe data of method NO.5 are all zeros, which
can not be taken into consideration. Table 5shows that our proposed method NO.6 has the
maximum total variance and the sum of thesquare distance between classes and method
NO.7 has the minimum sum of inner-class variances. It illustrates that our proposed methods NO.6 and NO.7 have advantages over GDA
and KPCA, quantitatively.

addition, the proposed method NO.6 receives
ation metric items. Meanwhile, except for the

Discussion on Observations from
Evaluation

total variance item, all of our proposed meth-Here, we make some analysis of the obser-

ods have better values than KPCA, quantita-

vation from the evaluation results. First, thetively.

evaluation performance of the Gaussian kernel

better values than KPCA on all of the evalu-

4.3– 10 –

original data

6

10.4
1
2
3

5

×10 8 KLPCDA Method No.1

10.2

4

KLPCDA Method No.2

1
1
2
3

10.3

1
2
3

450

0.5

10.1
400

Y

3

Y

Y

10

Y

Method No.3 GDA

500
1
2
3

0

9.9
2

350

9.8
-0.5

9.7

300

1
9.6
0
10

11

12

13

14

9.5
-1

15

-0.5

X

5

0

0.5

-1
-5

1

X

×10 9 Method No.4 KPCA

1KLPCDA Method No.5

1
2
3

4

1.5
1
2
3

0

5

X

×10 9

250
200

×10 8 KLPCDA Method No.6

600

×10 8 KLPCDA Method No.7
1
2
3

1

1
2
3

-4.0104-4.0104-4.0104

0

Y

Y

0.5

Y

500

-4.0104

3

Y

400

X

0.5

2

300

-4.0104-4.0104

01-4.0104

-0.5
-0.5

0

-4.0104-4.0104

-1
-1

-0.5

0

0.5

1

X

-1
-1

-0.5

0

0.5

-1
-5

1

X

0

5

X

×10 9

-1

-0.5

0

0.5

X

Fig. 4 Projections of seven kernelized data analysis methods on Wine data set by using Polynomial kernel with r = 3. The X axis is the ﬁrst principal component, and the Y axis is the second
principal component.
is better than that of the Polynomial kernel

and GDA. We will explore other combination

no matter conducting on the Fisher Iris data

ways according to the quantitative evaluation

set or Wine data set. While there comes one

results by contrasting the advantages and prop-

problem if the parameter σ = 0.2 of the Gaus-

erties of diﬀerent methods, to obtain more ef-

sian kernel is optimal, whether the polynomial

fective data analysis methods.

kernel performs inferior to the Gaussian ker-

Fourth, the projection directions and quan-nel with any parameter. We will work on thistitative data results can tell us the diﬀerences

parameter optimization problem in the future.

and changes in the inner relationships of data

Second, some of our proposed methods have

before and after being projected into higher

better performance and briefer deduction pro-dimensional space. It can help us to inquirecesses than GDA and KPCA under the same

into the possibility of kernelizing data analysis

conditions. It can be used to replace GDA or

methods in other spaces.

KPCA under certain circumstances for simpler
calculations. Moreover, some methods have so

5.

Conclusion

close projection directions that we can makeuse of to choose more suitable and simpler methods for diﬀerent problems.

In this paper, we extended the ﬁve data analysis methods proposed by Pei 19) named Lin-

Third, we proposed ﬁve methods from theear Principal Component Discriminant Analy-

linear combination of three objectives of KPCA

sis to high dimensional space by mapping thedata into a feature space and applying kernel– 11 –

1methods to handle mapped data. We called it3) Baudat, G.; and Anouar, F. 2000. Generalized
discriminant analysis using a kernel approach.
Neural computation, 12(10): 2385–2404.

Kernelized Linear Principal Component Discriminant Analysis.

It combines supervised4) Chin, T.-J.; and Suter, D. 2007. Incremental kernel principal component analysis.
IEEE transactions on image processing, 16(6):1662–1674.

and unsupervised learning methods so that we
can refer to our KLPCDA as semi-supervised
learning methods to be used for both classiﬁca-5) Cui, P.; Li, J.; and Wang, G. 2008. Improved
kernel principal component analysis for fault
detection. Expert Systems with Applications,34(2): 1210–1219.

tion and clustering problems. We evaluate twodata sets and use two kernel functions, fromwhich we illustrate the advantages of our pro-6) Ding, M.; Tian, Z.; and Xu, H. 2010. Adaptive
kernel principal component analysis. Signal
processing, 90(5): 1542–1553.

posed methods over KPCA and GDA.
In the future, we will continue to focus on7) Fisher, R. A. 1936. The use of multiple measurements in taxonomic problems. Annals of
eugenics, 7(2): 179–188.

solving three problems about our proposed
KLPCDA. The ﬁrst one is the parameter op-8) Fukunaga, K. 2013. Introduction to statistical
pattern recognition. Elsevier.

timization issue. We will explore the optimal
parameters of each kernel function resulting in9) Ghashami, M.; Perry, D. J.; and Phillips, J.
2016. Streaming kernel principal component
analysis. In Artiﬁcial intelligence and statistics, 1365–1374. PMLR.

the optimal evaluation results and attempt to
exhibit the changing trend of performance asparameters change visually. This is a significant work of practical application. The second one is investigating other possible combinations or creative ways of the three objectives
of KPCA and GDA, to obtain new kernelized
data analysis methods with lower complexity
and better classiﬁcation or clustering performance or any other advantages. The last one
is to apply our methods to the real scene, byusing more realistic data sets to solve morepractical problems and achieve more meaningful and eﬀective results. These subjects will be
involved in our future work.

10) Hofmann, T.; Schölkopf, B.; and Smola, A. J.
2008. Kernel methods in machine learning.
The annals of statistics, 36(3): 1171–1220.
11) Howland, P.; and Park, H. 2004. Generalizing discriminant analysis using the generalized
singular value decomposition. IEEE transactions on pattern analysis and machine intelligence, 26(8): 995–1006.
12) Izenman, A. J. 2013. Linear discriminant analysis. In Modern multivariate statistical techniques, 237–280. Springer.
13) Kim, K. I.; Franz, M. O.; and Scholkopf, B.
2005. Iterative kernel principal component
analysis for image modeling. IEEE transactions on pattern analysis and machine intelligence, 27(9): 1351–1366.

参考文献

14) Kim, K. I.; Jung, K.; and Kim, H. J. 2002.
Face recognition using kernel principal component analysis. IEEE signal processing letters,
9(2): 40–42.

1) Abdi, H.; and Williams, L. J. 2010. Principal
component analysis. Wiley interdisciplinary
reviews: computational statistics, 2(4): 433–
459.

15) Loh, W.-Y.; and Vanichsetakul, N. 1988. Treestructured classiﬁcation via generalized discriminant analysis. Journal of the American
Statistical Association, 83(403): 715–725.

2) Anderson, M. J.; and Robinson, J. 2003. Generalized discriminant analysis based on distances. Australian & New Zealand Journal of
Statistics, 45(3): 301–318.

16) Lu, J.; Plataniotis, K. N.; and Venetsanopoulos, A. N. 2003. Face recognition using kernel
direct discriminant analysis algorithms. IEEE
transactions on Neural Networks, 14(1): 117–
126.

– 12 –

17) Mika, S.; Schölkopf, B.; Smola, A.; Müller, K.R.; Scholz, M.; and Rätsch, G. 1998. Kernel
PCA and de-noising in feature spaces. Advances in neural information processing systems, 11.

classiﬁcation and correlation, M. Forina, R.
Leardi, C. Armanino and S. Lanteri, Elsevier,
Amsterdam, 1988, Price: US $645 ISBN 0444-43012-1. Journal of Chemometrics, 4(2):
191–193.

18) Nguyen, M.; and Torre, F. 2008. Robust kernel principal component analysis. Advances in
Neural Information Processing Systems, 21.

25) Wahba, G.; and Wang, Y. 2014. Representer
Theorem. Wiley StatsRef: Statistics Reference
Online, 1–11.

19) Pei, Y. 2015. Linear principal component discriminant analysis. In 2015 IEEE International Conference on Systems, Man, and Cybernetics, 2108–2113. IEEE.

26) Wold, S.; Esbensen, K.; and Geladi, P. 1987.
Principal component analysis. Chemometrics
and intelligent laboratory systems, 2(1-3): 37–
52.

20) Sarma, P.; Durlofsky, L. J.; and Aziz, K. 2008.
Kernel principal component analysis for eﬃcient, diﬀerentiable parameterization of multipoint geostatistics. Mathematical Geosciences,40(1): 3–32.

27) Ye, J.; Janardan, R.; Park, C. H.; and Park, H.
2004. An optimization criterion for generalized
discriminant analysis on undersampled problems. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 26(8): 982–994.

21) Schölkopf, B.; Smola, A.; and Müller, K.-R.
1998. Nonlinear component analysis as a kernel eigenvalue problem. Neural computation,
10(5): 1299–1319.

28) Ye, J.; and Yu, B. 2005. Characterizationof a family of algorithms for generalized discriminant analysis on undersampled problems.
Journal of Machine Learning Research, 6(4).

22) Stuhlsatz, A.; Lippel, J.; and Zielke, T. 2012.
Feature extraction with deep neural networks
by a generalized discriminant analysis. IEEE
transactions on neural networks and learning
systems, 23(4): 596–608.

29) Zhang, T.; Fang, B.; Tang, Y. Y.; Shang,Z.; and Xu, B. 2009. Generalized discriminant analysis: A matrix exponential approach.
IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 40(1): 186–
197.

23) Tipping, M. 2000. Sparse kernel principal
component analysis. Advances in neural information processing systems, 13.
24) Vandeginste, B. 1990. PARVUS: An extendable package of programs for data exploration,30) Zheng, W.; Zhao, L.; and Zou, C. 2004. A
modiﬁed algorithm for generalized discriminant analysis. Neural Computation, 16(6):1283–1297.

– 13 –