Local Fisher Discriminant Analysis
for Supervised Dimensionality Reduction

Masashi Sugiyama
sugi@cs.titech.ac.jp
Department of Computer Science, Tokyo Institute of Technology, 2-12-1-W8-74, O-okayama, Meguro-ku, Tokyo,152-8552, Japan

Abstract
Dimensionality reduction is one of the important preprocessing steps in high-dimensional
data analysis. In this paper, we consider thesupervised dimensionality reduction problem
where samples are accompanied with class labels. Traditional Fisher discriminant analysis is a popular and powerful method forthis purpose. However, it tends to give undesired results if samples in some class form
several separate clusters, i.e., multimodal. In
this paper, we propose a new dimensionality reduction method called local Fisher discriminant analysis (LFDA), which is a localized variant of Fisher discriminant analysis. LFDA takes local structure of the data
into account so the multimodal data can beembedded appropriately. We also show that
LFDA can be extended to non-linear dimensionality reduction scenarios by the kernel
trick.

1. Introduction
The goal of dimensionality reduction is to embed highdimensional data samples in a low-dimensional space
while most of ‘intrinsic information’ contained in thedata is preserved. Once dimensionality reduction iscarried out appropriately, we can utilize the compact
representation of the data for various succeeding tasks
such as visualization, classification, etc. In this paper,
we consider the supervised dimensionality reduction
problem where samples are accompanied with class labels.
Fisher discriminant analysis (FDA) (Fisher, 1936;
Fukunaga, 1990) is a popular method for linear dimenAppearing in Proceedings of the 23 rd International Conference on Machine Learning, Pittsburgh, PA, 2006. Copyright 2006 by the author(s)/owner(s).

sionality reduction1 , which maximizes between-classscatter and minimizes within-class scatter. This traditional FDA is known to work well and is practicallyuseful even now. However, it tends to give undesired
results if samples in some class form several separate
clusters (i.e., multimodal ) (Fukunaga, 1990).
Multimodality is often observed in many practical applications. For example, in disease diagnosis, the distribution of medial checkup samples of sick patients
could be multimodal since there may be several different causes even for a particular disease. Even ina traditional task of hand-written digit recognition,
multimodality appears if digits are classified into, e.g.,even and odd numbers. More generally, multimodality
can naturally appear if multi-class classification problems are solved by a set of two-class ‘one-versus-rest’
problems.
To embed multimodal data well, it is important to preserve local structure of the data. Locality-preserving
projection (LPP) (He & Niyogi, 2004) meets this requirement. LPP keeps nearby data pairs in the original
space close in the embedding space, by which multimodal data can be embedded without losing its local
structure. However, LPP is an unsupervised dimensionality reduction method which does not take thelabel information into account. Therefore, it does notnecessarily work appropriately in supervised dimensionality reduction scenarios.
In this paper, we propose a new dimensionality reduction method called local Fisher discriminant analysis
1

Usually, FDA may refer to a classification method
which first projects the data samples onto a onedimensional space and then classifies the samples bythresholding. The one-dimensional embedding space used
above is given as the maximizer of the so-called Fisher criterion. This Fisher criterion is often used for dimensionality reduction to a subspace with dimension more thanone (Fukunaga, 1990). With some abuse, we refer to thedimensionality reduction method based on the Fisher criterion as FDA in the following (see Section 2.2 for detail).

Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction(LFDA). LFDA combines the ideas of FDA and LPP:
between-class separability is maximized while withinclass local structure is preserved.

Later in Section 3.5, we discuss non-linear dimensionality reduction scenarios.

Because of the local structure preservation property,
multimodal labeled data can be effectively embedded
by LFDA. Furthermore, LFDA can give more separate embedding than FDA. To explain the reason intuitively, we first note that FDA can be regarded as
maximizing between-class scatter under the constraintof keeping within-class scatter to a certain level (Fukunaga, 1990). When samples in some class are multimodal, keeping within-class scatter to a certain levelappears to be quite hard since multimodal samples
should be typically merged into a single cluster. Dueto this strong constraint, less degree of freedom is left
for increasing separability, and thus FDA results in lessseparate embedding. On the other hand, LFDA does
not require multimodal samples to fall into a single
cluster. As a result, more degree of freedom is left forincreasing separability and thus highly separate embedding can be obtained.

2.2. Fisher Linear Discriminant

By the so-called kernel trick (Schölkopf & Smola,2002), FDA and LPP can be extended to non-linear
dimensionality reduction scenarios (Mika et al., 2003;
Belkin & Niyogi, 2003). We show that LFDA can also
be non-linearized by the kernel trick.

Here we briefly review the definition of Fisher criterion
for dimensionality reduction (Fisher, 1936; Fukunaga,
1990). With some abuse, we refer to the dimensionality reduction method based on the Fisher criterion as
Fisher discriminant analysis (FDA), where the original
FDA embeds the data samples only in one-dimensional
space.
Let S (w) and S (b) be the within-class scatter matrix
and the between-class scatter matrix defined by
S (w) =X̀ X

S (b) =X̀ni (µi − µ)(µi − µ)> ,(5)

where > denotes the transpose, µi is the mean of samples in the class i and µ is the mean of all samples:
µi =

1 X
xj ,
ni j:y =i

(6)

j

n

2. Linear Dimensionality Reduction

1X
xi .
n i=1

(7)

Using S (w) and S (b) , the FDA transformation matrix
T F DA is defined as follows:³´
T F DA = argmax tr (T > S (w) T )−1 T > S (b) T . (8)T ∈Rd×m

2.1. Formulation
Let xi ∈ Rd (i = 1, 2, . . . , n) be d-dimensional samples
and yi ∈ {1, 2, . . . , `} be associated class labels, where
n is the number of samples and ` is the number ofclasses. Let ni be the number of samples in the class
i:
P`
(1)
i=1 ni = n.
Let X be the matrix of all samples:
X = (x1 |x2 | · · · |xn ).

(4)

i=1

µ=
In this section, we formulate the problem of linear
dimensionality reduction and review typical existing
methods.

(xj − µi )(xj − µi )> ,i=1 j:yj =i(2)That is, T is determined so that between-class scatter
is maximized while within-class scatter is minimized.
It is known that T F DA is given by
T F DA = (ϕ1 |ϕ2 | · · · |ϕm ),(9)

where {ϕi }di=1 are the generalized eigenvectors associated to the generalized eigenvalues λ1 ≥ λ2 ≥ · · · ≥ λdof the following generalized eigenvalue problem:
S (b) ϕ = λS (w) ϕ.

(10)

m

Let z i ∈ R (1 ≤ m ≤ d) be embedded samples, wherem is the dimension of the embedding space. Effectively
we consider d to be large and m to be small, but notlimited to such cases.
For the moment, we focus on linear dimensionality reduction, i.e., using a d × m transformation matrix T ,z i is given byz i = T > xi .
(3)

2.3. Locality-Preserving Projection
Here we briefly review the definition of localitypreserving projection (LPP) (He & Niyogi, 2004). Let
A be the affinity matrix, i.e., the n-dimensional matrix
with the (i, j)-th element being the affinity between xi
and xj . We suppose that elements of A are in [0, 1]
and take larger values if xi and xj are ‘close’.

Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction

10

10

FDA
LPP
LFDA

10

FDA
LPP
LFDA

5

5

5

0

0

0

−5

−5

−5

−10
−10

−5

0

5

10

−10
−10

−5

0

(a)

5

10

−10
−10

FDA
LPP
LFDA

−5

(b)

0

5

10

(c)

Figure 1. Examples of dimensionality reduction by FDA, LPP and LFDA.

There are several different manners in defining A. A
simple one is to define Ai,j = 1 if xj is the k-nearest
neighbor of xi or vise versa; otherwise Ai,j = 0.
More elaborate methods can be found, e.g., in (He& Niyogi, 2004; Zelnik-Manor & Perona, 2005). The
latter may be useful for those who do not have any
subjective/prior preference.
Using A, the LPP transformation matrix T LP P is defined as follows.
n
1 X
T LP P = argmin
Ai,j kT > xi − T > xj k2
2
d×m
T ∈R
i,j=1
subject to T > XDX > T = I, (11)where I is the identity matrix and D is the ndimensional diagonal matrix with i-th diagonal element beingnX
D i,i =
Ai,j .
(12)j=1

Eq.(11) implies that in LPP, T is determined so thatnearby data pairs in the original space are kept closein the embedding space. Note that T > XDX > T = I
is a constraint to avoid a trivial solution T = O.
It is known that the LPP transformation matrix T LP P
is given by
T LP P = (ψ d−m+1 |ψ d−m+2 | · · · |ψ d ),(13)

where {ψ i }di=1 are the eigenvectors associated to theeigenvalues γ1 ≥ γ2 ≥ · · · ≥ γd of the following eigenvalue problem:
XLX > ψ = γXDX > ψ,

(14)

L = D − A.

(15)

where

2.4. Typical Behavior
In Figure 1, dimensionality reduction results obtained by FDA and LPP are shown, where twodimensional two-class data samples are embedded in a
one-dimensional subspace. In LPP, the affinity matrix
A is determined by the local scaling method (ZelnikManor & Perona, 2005).
For the simplest data set depicted in Figure 1(a), both
FDA and LPP give reasonable results where samples
of different classes are nicely separated. For the data
set depicted in Figure 1(b), FDA still works well. However, LPP mixes samples of different classes into one
cluster, which is caused by the unsupervised nature of
LPP. On the other hand, for the data set depicted in
Figure 1(c), LPP works well but FDA gives an undesired result. The reason is that the levels of betweenclass scatter and within-class scatter are not evaluatedin an intuitively natural way because of the two separate clusters of the same class. See also (Fukunaga,
1990).

3. Local Fisher Discriminant Analysis
As illustrated above, FDA can perform poorly if samples in some class form several separate clusters (i.e.,
multimodal). In other words, the undesired behavior
is caused by the globality when evaluating within-classscatter and between-class scatter. On the other hand,
LPP can make samples of different classes overlapped
if they are close in the original high-dimensional space.
To overcome this problem, we propose combining theidea of FDA and LPP: we evaluate the levels of withinclass scatter and between-class scatter in a local manner, by which class separability and local structureLocal Fisher Discriminant Analysis for Supervised Dimensionality Reduction

preservation could be attained at the same time. We
call our new method local Fisher discriminant analysis(LFDA).

Then we haven
Xn

To introduce our new method, let us first reformulate
FDA in a pairwise manner.

1 X
(w)
xi x>
j −S
n
i,j=1
i=1


n
n
n
X
X
X
1
1
(w)

=
xi x>
xi x>
i −
j −S
n
n
i=1
j=1
i,j=1

Lemma 1 S (w) and S (b) are expressed as

1 X=
2 i,j=1

S (b) =

3.1. Reformulating FDA

S (w) =n
1 X

2 i,j=1

Sn(w)

Ai,j (xi − xj )(xi − xj )> ,(16)

1 X (b)A (xi − xj )(xi − xj )> ,=2 i,j=1 i,j

(17)

1
(w)
− Ai,jn¶which yields Eq.(17).
Note that 1/n − 1/nc is negative while 1/nc and 1/n
are positive.
The above pairwise representation gives us a new interpretation of FDA: it tries to keep in-class data pairs(w)(b)
close (since Ai,j is positive and Ai,j is negative if

where

(b)

(
(w)
Ai,j =((b)

Ai,j =

1/nc
0

yi = yj ) and between-class data pairs apart (since Ai,jis positive if yi 6= yj ).

if yi = yj = c,(18)

if yi 6= yj ,1/n − 1/nc
1/n3.2. Definition of LFDA

if yi = yj = c,(19)

if yi 6= yj ,Based on the above pairwise expression, let us define(w)
the local within-class scatter matrix S
and the local(b)
between-class scatter matrix S as follows.
(w)(Proof ) It follows from Eq.(4) that

S


>
X
X
1
1
xj −
xp xj −xq S (w) =nniip:y =iq:y =i
i=1 j:y =i

S



X̀ X

p

j

=

µ

>
>
>
× (xi x>i + xj xj − xi xj − xj xi ), (22)

n

(b)

xi x>
i −

n
X

xi x>
i −

X̀ 1

q

X

xp x>
q

i=1

i=1

j=1

where

1 X (w)
A (xi − xj )(xi − xj )> ,2 i,j=1 i,j=

1 X (b)A (xi − xj )(xi − xj )> ,2 i,j=1 i,j

(
(

(b)
Ai,j =

i,j=1

(23)

n

(w)
Ai,j =

ni p,q:y =y =i
p
q


n
n
n
X
X
X
(w)
(w)

=
Ai,j  xi x>
Ai,j xi x>
i −
j
i=1

(b)

n

=

Ai,j /nc0

if yi = yj = c,if yi 6= yj ,Ai,j (1/n − 1/nc ) if yi = yj = c,1/nif yi 6= yj .

(24)

(25)
(26)

n

=

1 X (w)>>>
A (xi x>i + xj xj − xi xj − xj xi ),2 i,j=1 i,j(20)

which yields Eq.(16). Let S (m) be the mixture scatter
matrix (Fukunaga, 1990):
S (m) = S (w) + S (b)nX=(xi − µ)(xi − µ)> .
i=1

Compared with the global counterparts S (w) and S (b) ,the values for in-class pairs are weighted by the affinity(w)(b)
and S . This means that far-apart in-class
in S
(w)
(b)
pairs have less influence in S
and S .
(w)(b)

Using Sand S , we define the LFDA transformation matrix T LF DA as
³
´
(w)
(b)
T LF DA = argmax tr (T > S T )−1 T > S T .
T ∈Rd×m(21)(27)
That is, we determine T so that nearby data pairs of
the same class are close and data pairs of different

Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction

classes are apart. Data pairs of the same class but far
apart are not imposed to be close.
If the affinity matrix A is taken to be one for all inclass pairs (i.e., all in-class pairs are ‘equally close’ each
(w)

(b)

other), S
and S
are reduced to S (w) and S (b)so LFDA is reduced to the ordinary FDA. Therefore,
LFDA may be regarded as a natural localized variantof FDA.
Since Eq.(27) is the same form as Eq.(8), we can easily obtain T LF DA by solving a generalized eigenvalue
problem. In Section 3.4, we show an efficient method
for computing T LF DA .
Examples of dimensionality reduction by LFDA are
illustrated in Figure 1, where the affinity matrix A is
determined by the same way as being done for LPP.
The figures show that LFDA gives desirable results
for all three data sets, i.e., LFDA can compensate thedrawbacks of FDA and LPP.
3.3. Why is LFDA good?

The constraint in Eq.(29) is less restrictive than thatin Eq.(28) since faraway pairs of in-class samples are
not imposed to be close. Due to this weaker constraint,
the degree of freedom which can be used for maximizing separability between different classes remains morethan FDA. Thus, LFDA can result in more separate
embedding.
3.4. Efficiently Computing LFDA
Transformation Matrix
Here, we provide an efficient method to compute the
LFDA transformation matrix T LF DA .
Let S
by

(m)

S

(m)

be the local mixture scatter matrix defined

=S

(w)

(b)

+S
n
X
1
(m)
=
A (xi − xj )(xi − xj )> ,2 i,j=1 i,j

(30)

(m)

LFDA does not impose far-apart data pairs of the sameclass to be close, by which local structure of the data
tends to be preserved. This itself is a useful property,
e.g., in data visualization tasks because ‘interesting’
structure such as multimodality is not lost by dimensionality reduction.

is the n-dimensional matrix with (i, j)-thwhere Aelement being

In addition to that, LFDA can even provide more separate embedding than FDA. To explain the reason, let
us recall that the FDA transformation matrix T F DA
can also be obtained as follows (Fukunaga, 1990).
³´
T F DA = argmax tr T > S (b) T

Then the same solution T LF DA can be still obtained(b)(m)
if S in Eq.(27) is replaced by S
because of thefollowing identity (cf. (Fukunaga, 1990)):

T ∈Rd×m

subject to T > S (w) T = I.

(28)

This representation implies that FDA maximizes
between-class scatter under the constraint of keeping
within-class scatter to a certain level.
When one of the classes is multimodal, the above constraint is actually quite restrictive since the multimodal data should be typically merged into a single
cluster. Therefore, the remaining degree of freedom
which can be used for maximizing between-class scatter may not be that much. As a result, FDA can resultin less separate embedding.
On the other hand, it is straightforward to show that
T LF DA can also be expressed as follows.
³´(b)
T LF DA = argmax tr T > S T
T ∈Rd×m

subject to T > S(w)

T = I.

(29)

(
(m)
(w)
(b)
Ai,j = Ai,j + Ai,j =Ai,j /n if yi = yj ,1/nif yi 6= yj .

³
´
(w)
(m)
tr (T > S T )−1 T > S T
³
´
(w)
(b)
= tr (T > S T )−1 T > S T + m.

(31)

(32)

In the following, we employ the criterion with S
since it is simpler.

(m)

The left-hand side of Eq.(32) is the same form as FDA.
Therefore, the solution T LF DA is analytically given as
follows.
T LF DA = (ϕ1 |ϕ2 | · · · |ϕm ),(33)

where {ϕi }di=1 are the generalized eigenvectors associated to the generalized eigenvalues λ1 ≥ λ2 ≥ · · · ≥ λdof the following generalized eigenvalue problem:
S

(m)

ϕ=λS

Below, we show how S
efficiently.

(m)

(w)

and S

ϕ.

(w)

(34)
can be computed

Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction

Since
S

(m)

n
X

1
(m)
>
>
>
A (xi x>i + xj xj − xi xj − xj xi )2 i,j=1 i,j


n
n
n
X
X
X
(m)
(m)

=
Ai,j  xi x>
Ai,j xi x>
i −
j ,
=

i=1

j=1

i,j=1

(35)
S

(m)

can be expressed in a matrix form as
S

(m)

(m)

= XL

(m)

(m)

X >,

(36)

(m)

(m)

= D
−A
and D
is the nwhere L
dimensional diagonal matrix with i-th diagonal elePn
(m)
(m)
(w)
ment being D i,i = j=1 Ai,j . Similarly, S
canbe expressed in a matrix form as
S
(w)

(w)

= XL

(w)

(w)

X >,

(37)

(w)

(w)

where L
= D
− A
and D
is the ndimensional diagonal matrix with i-th diagonal elePn(w)(w)
ment being D i,i = j=1 Ai,j .
(m)

(w)

L
and L
are n-dimensional matrices so could(w)
be very high dimensional. However, L
could be(w)
sparse so computing S
directly by Eq.(37) may be(m)efficient. On the other hand, computing S
directly(m)by Eq.(36) is not so efficient since A
is dense.
This problem can be alleviated as follows. A
be decomposed asA(m)= 11> /n + A

(m0 )

(m)

,

can
(38)

where
1 is the n-dimensional vector with all ones and(m0 )A
is the n-dimensional matrix with (i, j)-th element being((Ai,j − 1)/n if yi = yj ,(m0 )Ai,j =(39)
0
if yi 6= yj .
Then SS(m)(m)

can be expressed as follows:= XD

where D(m)(m)

X > − (X1)(X1)> /n − XA

is expressed by using A(m)

D i,i = 1 +

n
X

(m0 )

(m0 )

Ai,j .

(m0 )

X >,
(40)

as
(41)

j=1
(m0 )

A
becomes a block-diagonal matrix if {xi }ni=1 aresorted by the labels, which implies that the third termin the right-hand side of Eq.(40) may be computed efficiently. Since the first two terms in the right-hand sideof Eq.(40) can also be computed efficiently, computing(m)S
by Eq.(40) may be more efficient than directly
by Eq.(36).
To further improve computational efficiency, the affinity matrix A may be computed in a classwise manner,i.e., the between-class affinity is set to be zero.
3.5. Kernel LFDA for Non-Linear
Dimensionality Reduction
So far, we focused on linear dimensionality reduction.
Here we extend our discussion to non-linear dimensionality reduction scenarios.
From Eqs.(36) and (37), the generalized eigenvalue
problem (34) can be expressed as
XL(m)

X > ϕ = λXL(w)

X > ϕ.

(42)

When d ≤ n, any vector ϕ ∈ Rd can be expressedby using some vector α ∈ Rn as ϕ = Xα. Then,
multiplying Eq.(42) by X > from the left-hand side,
we have(m)(w)
KL Kα = λKL Kα,(43)
where K is the n-dimensional matrix with the (i, j)-thelement being
K i,j = x>(44)
i xj .
This implies that {xi }ni=1 appear only in terms of theirinner products, so we can non-linearize the algorithm
by the kernel trick (Schölkopf & Smola, 2002), whichis briefly explained below.
Let us consider a non-linear mapping φ(x) from Rd toa reproducing kernel Hilbert space H. Let K(x, x0 ) be
the reproducing kernel of H. A typical choice of thekernel function would be the Gaussian kernel:¡¢
K(x, x0 ) = exp −kx − x0 k2 /2σ 2 .
(45)
For other choices, see, e.g., (Schölkopf & Smola, 2002).
Because of the reproducing property of K(x, x0 ), K
is now the kernel matrix, i.e., the (i, j)-th element isdefined as
K i,j = hφ(xi ), φ(xj )i = K(xi , xj ),(46)

where h·, ·i denotes the inner product in H. Then theembedded image of φ(x0 ) by LFDA in H is given byK(x1 , x0 )
 K(x2 , x0 ) 
(α1 |α2 | · · · |αm )> 
(47)
,
..


.
K(xn , x0 )

Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction

where {αi }di=1 is the generalized eigenvectors associated to the generalized eigenvalues λ1 ≥ λ2 ≥ · · · ≥ λdof the generalized eigenvalue problem (43). Note that
KL
KL

(m)

(m)

K can be efficiently calculated by
K = KD(m)

K − (K1)(K1)> /n − KA

(m0 )

K.
(48)

4. Numerical Results
In this section, we apply the existing and proposed
dimensionality reduction methods to benchmark data
sets for visualization. In LPP and LFDA, the affinity matrix is determined by the same method used in
Section 2.4.
We use Iris, Letter recognition, Segment, and Thyroid disease data sets available from the UCI machinelearning repository. They are all multi-class classification data sets. We created two-class problems in thestandard ‘one-versus-rest’ manner. Figure 2 shows theembedded samples in two-dimensional space by FDA,
LFDA, and LPP.
For Iris data set, FDA tends to mix samples of different classes, while LFDA separates them very well.
Multimodality of the ‘×’-class can be clearly observedin the results of LFDA, which implies that FDA did
not work well because of multimodality. This experimental result supports the qualitative justification of
LFDA given in Section 3.3. LPP also works well forthis data set since three clusters (two ‘×’-class clusters
and one ’◦’-class cluster) are well separated from eachother in the original space.
For the other three data sets, FDA can separate samples of different classes quite well. LFDA also separates them equally well, but moreover it preserves
multimodality of the ‘×’-class more prominently than
FDA. This would be a desirable property in data visualization tasks. LPP also preserves multimodality ofthe data well, but it tends to mix samples of different
classes.
Overall, LFDA is found to be useful in data visualization tasks.

5. Conclusions
Dimensionality reduction based on the Fisher criterion(FDA) works well given data samples of each classform a single cluster (i.e., unimodal). On the other
hand, samples in some class can be multimodal, e.g.,when multi-class classification problems are solved bya set of two-class ‘one-versus-rest’ problems. In thispaper, we first showed that FDA can be regarded

as keeping in-class data pairs close and between-classdata pairs apart. Based on this novel interpretation, we proposed a localized variant of FDA called
local Fisher discriminant analysis (LFDA). We experimentally showed that LFDA can preserve multimodal
structure of the data better than FDA, and LFDA caneven provide more separate embedding than FDA.
Our future work includes the comparison with recently proposed methods, e.g, (Goldberger et al., 2005;
Globerson & Roweis, 2006).

Acknowledgments
The author would like to thank Yuki Shinada, KlausRobert Müller, Hideki Asoh, Stefan Harmeling andanonymous reviewers for their comments. He also acknowledges financial support from MEXT (Grant-inAid for Young Scientists 17700142).

References
Belkin, M., & Niyogi, P. (2003). Laplacian eigenmaps for
dimensionality reduction and data representation. Neural Computation, 15, 1373–1396.
Fisher, R. A. (1936). The use of multiple measurements intaxonomic problems. Annals of Eugenics, 7, 179–188.
Fukunaga, K. (1990). Introduction to statistical patternrecognition. Boston: Academic Press, Inc. Second edition.
Globerson, A., & Roweis, S. (2006). Metric learning bycollapsing classes. Advances in Neural Information Processing Systems 18 (pp. 451–458). Cambridge, MA: MIT
Press.
Goldberger, J., Roweis, S., Hinton, G., & Salakhutdinov,
R. (2005). Neighbourhood components analysis. In L. K.
Saul, Y. Weiss and L. Bottou (Eds.), Advances in neural
information processing systems 17, 513–520. Cambridge,
MA: MIT Press.
He, X., & Niyogi, P. (2004). Locality preserving projections. In S. Thrun, L. Saul and B. Schölkopf (Eds.),
Advances in neural information processing systems 16.
Cambridge, MA: MIT Press.
Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Smola, A.,& Müller, K.-R. (2003). Constructing descriptive anddiscriminative nonlinear features: Rayleigh coefficientsin kernel feature spaces. IEEE Transactions on PatternAnalysis and Machine Intelligence, 25, 623–628.
Schölkopf, B., & Smola, A. J. (2002). Learning with kernels. Cambridge, MA: MIT Press.
Zelnik-Manor, L., & Perona, P. (2005). Self-tuning spectralclustering. In L. K. Saul, Y. Weiss and L. Bottou (Eds.),
Advances in neural information processing systems 17,
1601–1608. Cambridge, MA: MIT Press.

Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction
FDA

1.2

LFDA

4

1

3

LPP

1.5

SetosaVirginica
Verisicolour

1

0.8

2

0.6

0.5

0.4

1

0.2

0

0

0

−0.5

−0.2

−1

−0.4

−1

−2
−0.6
−0.8
−1

−0.5

0

0.5

1

FDA

5

−3
−2

−1

−0.5

4

3

3

2

2

1

1

0

0.5

1

1.5

−1.5
−4

−3

−2

−1

0

LFDA

5

4

15

A
C
B

1

2

3

4

4

6

LPP

10

5

0

0

−1

−1

−2

−2

−3

−3

−4

−4

−5
−2

−1.5

−1

−0.5

0

0.5

1

1.5

0

−5

−10

−5
−8

−6

−4

−2

0

2

4

6

−15
−8

−6

−4

−2

FDA

−8

8

−1.5

x 10

5

6

BrickfaceSky
Foliage

4

2

LPP

−7

2

LFDA

−7

x 10

0

x 10

1.5
1

4
3

0.5

2
2

0

0
1

−2

−4

−6
−4

−3

−2

−1

0

1

2

3

−9

−0.5

0

−1

−1

−1.5

−2
−5

0

5

10

15

20

−7

x 10
FDA

15

LFDA

50
40

10

−2
−3

−2

−1

0

1

2

3

4

x 10

LPP

30

Hyper
Hypo
Normal

5

−7

x 10

25

30

20

20

15

10

10

0

5

−10

0

5

0

−5

−10
−6

−4

−2

0

2

4

−20

−5

−30

−10

−40
−10

−5

0

5

10

15

20

25

−15
−40

−20

0

20

40

60

Figure 2. Results of data visualization (m = 2). From top to bottom, Iris data set (d = 4: ‘Setosa’ & ‘Virginica’vs. ‘Versicolour’), Letter recognition data set (d = 16: ‘A’ & ‘C’ vs. ‘B’), Segment data set (d = 18: ‘Brickface’ & ‘Sky’vs. ‘Foliage’), and Thyroid disease data set (d = 5: ‘Hyper’ & ‘Hypo’ vs. ‘Normal’). From left to right, FDA, LFDA, and
LPP.