# A Unified Factors Analysis Framework for Discriminative Feature Extraction and Object Recognition

Authors: Ningbo Hao; Jie Yang; Haibin Liao; Wenhua Dai
Journal: Mathematical Problems in Engineering
 (2016)

---

## Abstract

Various methods for feature extraction and dimensionality reduction have been proposed in recent decades, including supervised and unsupervised methods and linear and nonlinear methods. Despite the different motivations of these methods, we present in this paper a general formulation known as factor analysis to unify them within a common framework. During factor analysis, an object can be seen as being comprised of content and style factors, and the objective of feature extraction and dimensionality reduction is to obtain the content factor without style factor. There are two vital steps in factor analysis framework; one is the design of factor separating objective function, including the design of partition and weight matrix, and the other is the design of space mapping function. In this paper, classical Linear Discriminant Analysis (LDA) and Locality Preserving Projection (LPP) algorithms are improved based on factor analysis framework, and LDA based on factor analysis (FA-LDA) and LPP based on factor analysis (FA-LPP) are proposed. Experimental results show the superiority of our proposed approach in classification performance compared to classical LDA and LPP algorithms.


---

## Body

## 1. Introduction

Feature extraction and selection are both a critical and challenging component of pattern recognition [1, 2]. Some basic principles can be used to guide the design of a feature extractor; however, the design work is essentially an experimental science depending on its field and problem being addressed. The utilization of algorithms for dimensionality reduction in supervised or unsupervised learning tasks has attracted a lot of attention in recent years. The relative simplicity and effectiveness of the linear algorithms for both Principal Component Analysis (PCA) [3] and Linear Discriminant Analysis (LDA) [3, 4] have made these two algorithms very attractive. The Locality Preserving Projections (LPP) [5] and Discriminant Canonical Correlations (DCC) [6] are other linear algorithms that have been proposed for dimensionality reduction. Three other recently developed algorithms—termed ISOMAP [7], LLE [8], and Laplacian Eigenmap [9]—can be utilized when conducting nonlinear dimensionality on a dataset that lies on or around a lower-dimensional manifold. Additionally, in order to extend the linear dimensionality reduction algorithms to nonlinear ones, the kernel trick approach [10] has also been applied to perform linear operations on higher or infinite dimensional features which are transformed by a kernel mapping function. What is more, a number of algorithms [11–14] have recently been proposed to carry out dimensionality reduction on objects encoded as matrices or tensors of an arbitrary order.All the aforementioned algorithms have been designed according to specific intuitions. Solutions are given through the optimization of both intuitive and pragmatic objectives, with the aim of extracting low-dimensional, simplified, and robust features from the high-dimensional, redundant, disturbed, and distorted original data. Content and style factors can be seen as the two components of an object [15]. During pattern recognition field, the information explored as base of recognition is defined as content factor and the information disturbed recognition is defined as style factor. Two examples are given below. One is for face recognition. A front face with neutral expression, no occlusion, and normalized illumination represents the content factor of a face image, whereas a face with variation of poses, illumination, and expressions represents the style factor, as shown in Figure 1. The occurrence of interference between content and style factors necessitates their separation in order to achieve accurate classification capabilities. The other is for speech recognition. In the process of recognizing the text of speech, the style factor includes timbre, tone, and emotion of the speaker. These elements should be regarded as background noise and minimized, as they do not provide any useful information related to speech text. However, the situation is opposite for speaker recognition since the favorable element is timbre information, which is defined as content factor here. The influence of speech text which is defined as style factor should be minimized. Thus the primary objective of the linear, nonlinear, manifold, or tensor decomposition methods is to reduce the style factors of the object.Figure 1
Face factors analysis (the content factors which without illumination and poses are what we want to extract and the style factors which with illumination and poses are what we want to remove).
(a)
(b)In this paper, we present a general framework called factors analysis, along with its linearization, kernelization, and tensorization, which offers a unified view for understanding and explaining many of the popular dimensionality reduction algorithms such as the ones mentioned above. The main objective of factor analysis is to find the essential low-dimensional features by simultaneously minimizing the style factors and maximizing the content factors of the object through the method of projection. There are two vital steps in the factor analysis framework. The first is to construct a factor separating objective function which includes the construction of a partition and weight matrix. The second step is to design the space mapping function. Using the factor analysis framework as a platform, we develop two novel dimensionality reduction algorithms, FA-LDA and FA-LPP. FA-LDA is to overcome the limitations of LDA, which solves the problem that the intraclass and interclass scatter matrix in LDA are not optimal through setting proper weights. FA-LPP is to overcome the limitations of LPP, which improves the design of partition so that the objects not only preserve the locality similarity but also have abundant identification information.


## 2. Factors Analysis: A General Framework for Feature Extraction

### 2.1. Factors Analysis

Content and style factors can be seen as two independent elements which influence the object and determine the observation [15], as mentioned before. The information contributed to recognition is defined as content factor and the negative information is style factor during pattern recognition.For a general classification problem, the sample set is represented asX
=
[
x
1
,
x
2
,
…
,
x
N
], x
i
∈
R
m, where N denotes the sample number and m denotes the feature dimension. In practice, the feature dimension m is very high usually, including redundancy and interference information. In order to alleviate the curse of dimensionality, it is necessary to transform the original data X from a high-dimensional space to a low-dimensional space which is represented as Y
=
[
y
1
,
y
2
,
…
,
y
N
]
,
y
i
∈
R
m
′. The essence of dimensionality reduction is to find a mapping function F
:
x
→
y
^ that transforms x
∈
R
m into appropriate low-dimensional space y
^
∈
R
m
′, where typically m
≫
m
′:(1)
y
^
=
F
x
.According to the various cases, the mapping function can be explicit or implicit and linear or nonlinear. A dimensionality reduction algorithm based on factor analysis is proposed in this paper, which is different from traditional methods. The objective of the dimensionality reduction algorithm based on factor analysis is to find a low-dimensional datasetY from the original dataset X, in which the style factor is suppressed as much as possible. The objective function is represented as follows:(2)
y
∗
=
arg
⁡
max
⁡
F
c
′
F
s
′
=
arg
⁡
max
⁡
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
μ
i
′
-
μ
j
′
2
W
i
j
c
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
y
j
i
-
y
k
i
2
W
j
k
s
=
arg
⁡
min
F
c
′
=
d
⁡
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
y
j
i
-
y
k
i
2
W
j
k
s
,where F
c
′, F
s
′ are the content and style variance matrices after mapping (factors separating), respectively, C is the total number of partitions, N
i is the sample number of the i
t
h partition, p
i is the prior probability of the i
t
h partition (including the i
t
h class for supervised methods and the i
t
h local manifold space for unsupervised methods, generally named as the i
t
h partition), μ
i
′ is the mean vector of the i
t
h partition after factors separating, y
j
(
i
), y
k
(
i
) are the j
t
h and k
t
h sample vectors of the i
t
h partition, respectively, after factors separating, W
i
j
c, W
j
k
s are weight matrices of content variance and style variance, respectively, and d is the constraint constant, which is different in different cases. Factor analysis criteria make the content variance of the objects between partitions as high as possible and the style variance of the objects within a partition as low as possible. In order to eliminate the style factor of the objects maximally, the style factor should be minimized and the content factor should be maximized after mapping transformation.


### 2.2. General Framework for Feature Extract

There are two vital steps for feature extraction based on factor analysis framework; one is the design of factor separating objective function, mainly including the design of partition and weight matrix, and the other is the design of space mapping function. Partition is designed based on the class information with supervised methods or local neighborhood through unsupervised or semisupervised methods of the samples. Weight matrix is designed according to the classification performance for purpose of finding the optimal distribution scheme, which is biased to the weakly separated samples in the case without sacrificing the strongly separated samples. Space mapping function is designed to transform the samples from input space to feature space. Thus, the research of factor analysis framework can proceed from the following two aspects: the first is to propose different space mapping transformation aimed at different applications, and the second is to find the design scheme of factor separating objective function that matches the pattern recognition best.


#### 2.2.1. Space Mapping Transformation

There are three categories of space mapping transformation algorithms, which are linear space, kernel space, and tensor space mapping transformation, respectively.(a) Linear Space Mapping Transformation. The relationship of datasets X and Y is supposed to be linear in linear space mapping transformation; namely, y
i
=
ψ
T
x
i. Thus the objective function (see (2)) is converted to(3)
J
ψ
=
arg
⁡
max
⁡
trace
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
ψ
T
μ
i
0
-
μ
j
0
W
i
j
c
μ
i
0
-
μ
j
0
T
ψ
trace
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
ψ
T
x
j
i
-
x
k
i
W
j
k
s
x
j
i
-
x
k
i
T
ψ
=
arg
⁡
max
⁡
trace
ψ
T
F
c
ψ
trace
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
trace
ψ
T
F
s
ψ
,where μ
i
0 is the mean vector of the i
t
h partition in original space, x
j
(
i
), x
k
(
i
) are the j
t
h and k
t
h sample vectors of the i
t
h partition samples in original space, and F
c, F
s are the content and style variance matrices in original space, respectively, defined as follows:(4)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
W
i
j
c
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
.(b) Kernel Space Mapping Transformation. The direct way to extend methods from linear projections to nonlinear cases is to utilize the kernel trick. The intuition of the kernel methods is to map the data from the original input space to a higher-dimensional Hilbert space (kernel space) as ϕ
:
x
→
H. The content and style variance matrices F
c, F
s in kernel space H are defined as follows:(5)
F
c
ϕ
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
W
i
j
c
μ
i
0
ϕ
-
μ
j
0
ϕ
μ
i
0
ϕ
-
μ
j
0
ϕ
T
,
F
s
ϕ
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
ϕ
x
j
i
-
ϕ
x
k
i
ϕ
x
j
i
-
ϕ
x
k
i
T
,where μ
i
0
ϕ
=
1
/
N
i
∑
i
=
1
N
i
ϕ
(
x
i
0
) is the mean vector of the i
t
h partition samples in kernel space H and ϕ
(
x
j
(
i
)
) is the kernel mapping to x
j
(
i
).(c) Tensor Space Mapping Transformation. Tensorization is another extension of linear methods, and it is a multiple linear extension. Firstly some terminologies [16] on tensor operations are reviewed before introducing the tensor space mapping transformation.Tensor is the extension of vector and matrix. TensorA
∈
R
I
1
×
I
2
×
⋯
×
I
r represents m
r components in m-dimension space, where every component is a function of coordinates, linearly changing with the coordinates following certain rules. r is the order of the tensor. When r
=
1
A is simplified to a vector and when r
=
2
A is simplified to a matrix. Expanding the tensors can facilitate its matrix representation. The k-order expansion of tensor A is A
(
k
)
∈
R
I
k
×
(
I
k
+
1
⋯
I
r
I
1
I
2
⋯
I
r
-
1
). The k-mode product between a tensor A and a matrix U
∈
R
J
k
×
I
k is defined as B
=
A
×
k
U, or B
(
k
)
=
U
A
(
k
) for the form of tensor expansion.Reform the original high-dimensional samples as the tensor{
X
i
∈
R
I
1
×
I
2
×
⋯
×
I
r
,
i
=
1,2
,
…
,
N
}. Corresponding low-dimensional tensors as (6) are obtained from high-dimensional tensors through projecting (multiple linear projection), which is similar to linear methods:(6)
y
i
=
X
i
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
.Then the objective function (see (2)) is converted to (7)
ψ
1
,
…
,
ψ
r
∗
=
arg
⁡
max
⁡
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
μ
i
0
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
-
μ
j
0
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
2
W
i
j
c
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
X
j
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
-
X
k
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
2
W
j
k
s
.


#### 2.2.2. The Design of Factor Separating Objective Function

The design of factor separating objective function is mainly about separating the content and style factors of objects, including the design of partitions and weight matrix.There are two schemes for partition design recently; one is based on supervised algorithms, such as LDA, in which the samples are partitioned according to the class information of samples; the other is based on unsupervised algorithms, such as LPP, in which the samples are partitioned according to thek-nearest neighborhood of samples.Variant weighting schemes have been proposed for different cases, which can be classified into four categories according to the weighting algorithms for scatter matrix between classes (content variance matrixF
c) in supervised face recognition methods based on weighted Linear Discriminant Analysis [17].(1) Euclidean Distance Algorithm [17]. Consider(8)
w
i
,
j
=
1
e
d
2
,where e
d is the Euclidean distance between the mean vectors of two classes.(2) Estimating Bayesian Error Rate Algorithm [18]. Consider(9)
w
i
,
j
=
1
2
Δ
i
j
2
erf
⁡
Δ
i
j
2
2
,where Δ
i
j is the Mahalanobis distance between the mean vectors of two classes and erf
⁡
(
x
)
=
2
/
π
∫
0
x
e
-
t
2
d
t is the error function.(3) Confusing Information Algorithm [19]. Consider(10)
w
i
,
j
=
E
i
j
n
i
if
i
≠
j
0
if
i
=
j
,where E
i
j is the number of samples which are misclassified into class j but belong to class i actually and n
i is the sample number of the i
t
h class.(4) Grey Correlation Analysis Algorithm Proposed in This Paper. Consider(11)
w
i
,
j
=
1
n
∑
k
=
1
n
γ
μ
i
k
,
μ
j
k
,where n is the vector dimensionality and μ
i
(
k
), μ
j
(
k
) are the k
t
h feature in the mean vectors of the i
t
h and j
t
h partitions. γ
(
μ
i
(
k
), μ
j
(
k
)
) is the correlation coefficients of the k
t
h feature in the mean vectors of the i
t
h and j
t
h partitions, defined as follows: (12)
γ
μ
i
k
,
μ
j
k
=
min
j
⁡
min
k
⁡
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
.The weighting scheme about the style variance matrixF
s gets little attention recently. According to the weighting scheme in LPP method, a new weighting scheme for F
s based on thermonuclear similarity is proposed in this paper:(13)
W
i
j
=
exp
⁡
x
i
-
x
j
2
2
t
x
i
,
x
j
belong  to  same  partition
0
otherwise.It deserves deep research on how to distribute the weights effectively and reasonably. The optimal weighting scheme will never abandon any object that may be classified correctly and never waste any weight value. In other words, it will help all the objects which need help as much as possible in order to achieve a win-win situation.


### 2.3. Unifying Various Algorithms of Feature Extraction

The relationship between several classical subspace dimensionality reduction algorithms and factor analysis framework is discussed in this section. Actually, most subspace algorithms that are commonly used recently can be represented and explained by factor analysis framework. After unifying current subspace algorithms in framework, it is obvious that these algorithms are just different in the basis of factor construction and constraint of factor separation, though the motivation or objective of these algorithms is varying. Next we will introduce the relationships between factor analysis and some typical methods of dimension reduction based on subspace.


#### 2.3.1. The Relationship with LDA

The objective of LDA is to find the optimal projection direction of classification that maximizes the ratio between the intraclass and interclass scatter matrix:(14)
J
ψ
=
arg
⁡
max
⁡
trace
ψ
T
S
b
ψ
trace
ψ
T
S
w
ψ
,
(15)
S
b
=
∑
i
=
1
C
p
i
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
,
(16)
S
w
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
x
j
0
-
μ
i
0
x
j
0
-
μ
i
0
T
,where C is the class number, p
i is the prior probability of class i, μ
i
0 is the mean vector of class i before projection, μ
0 is the overall mean vector before projection, and N
i is the sample number of class i. Refer to (3) and (4), set the weight matrix of F
c and F
s as all 1 matrix, and then(17)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
.After further conversion,(18)
F
c
=
∑
i
=
1
C
p
i
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
=
S
b
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
x
j
0
-
μ
i
0
x
j
0
-
μ
i
0
T
=
S
w
.Substitute (17) into (14); then(19)
J
ψ
=
arg
⁡
max
⁡
trace
ψ
T
S
b
ψ
trace
ψ
T
S
w
ψ
=
trace
ψ
T
F
c
ψ
trace
ψ
T
F
s
ψ
.Thus, LDA is a particular case of factor analysis framework when the entire weight matrix is 1 and the basis of partition is class information. The content factor is the accumulation of mean variance among each class and the style factor is the variance of each sample in the same class.


#### 2.3.2. The Relationship with MFA [20] and DLA [21]

In order to overcome the hypothesis of LDA that the samples should be of Gaussian distribution, MFA redefines the scatter matrix within class and between classes of LDA, and the objective function is as follows:(20)
J
ψ
=
arg
⁡
max
⁡
∑
c
∑
i
,
j
∈
P
k
2
c
or
j
,
i
∈
P
k
2
c
′
ψ
T
x
i
c
-
ψ
T
x
j
c
′
2
∑
c
=
1
C
∑
j
∈
N
K
1
c
i
or
i
∈
N
k
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
,where c is the class, P
k
2
(
c
) is a data pairs set that are the k
2-nearest pairs between two different classes c and c
′, and N
k
1
c
(
i
) indicates the index set of the k
1-nearest neighbors of the sample i within the c
t
h class.It can be concluded that MFA is also a particular case of factor analysis framework; only it considers both the class information and nearest neighborhood of the samples for the partition design. The numerator of (20) can be rewritten as follows:(21)
∑
c
∑
i
,
j
∈
P
k
2
c
or
j
,
i
∈
P
k
2
c
′
ψ
T
x
i
c
-
ψ
T
x
j
c
′
2
=
∑
i
=
1
C
-
1
∑
j
=
1
C
ψ
T
μ
i
0
-
ψ
T
μ
j
0
2
=
ψ
T
F
c
ψ
,where μ
i
0, μ
j
0 are the mean vector of the k
2-nearest pairs between the i
t
h and j
t
h classes. Therefore, the interclass separability can be seen as the content factor in factor analysis framework. Though the class information and nearest neighborhood are considered in the design of interclass separability, the weight matrix is ignored.Similarly, the denominator of (20) can be rewritten as follows:(22)
∑
c
=
1
C
∑
j
∈
N
K
1
C
i
or
i
∈
N
k
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
=
∑
c
=
1
C
∑
i
=
1
N
K
1
c
i
∑
j
=
1
N
K
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
=
ψ
T
F
s
ψ
.Therefore, the intraclass compactness of MFA can be seen as the style factor in factor analysis framework. Interclass separability is conducted based on thek-nearest neighbor relationships of the samples from the same class, disregarding the design of weight matrix.For DLA, the design criterion of part optimization is that the distance of the nearest neighbor image is minimized within class and maximized among classes. The objective function of DLA is equal to (20), and thus DLA is a particular case of factor analysis framework also.


#### 2.3.3. The Relationship with LPP

Refer to (3); J
(
ψ
)
=
arg
⁡
m
i
n
ψ
T
F
c
ψ
=
d
trace
ψ
T
F
s
ψ. Assume that all the prior probability p
i is identical, and then the content and style variance matrix can be represented as follows:(23)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
W
i
j
c
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
=
∑
i
=
1
C
W
i
j
c
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
,
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
,where F
s can be represented as follows:(24)
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
2
.Partition the samples according to k-nearest neighbors; then (24) can be written as(25)
F
s
=
∑
i
N
∑
j
N
x
i
-
x
j
2
w
i
j
,where W
∈
R
N
×
N is the relation matrix weighted by the heat kernels: w
(
i
,
j
)
=
exp
⁡
⁡
(
-
x
i
-
x
j
2
/
t
) if x
i is one of the k-nearest neighbors of x
j or x
j is one of the k-nearest neighbors of x
i; otherwise it is 0, and t is a tuning parameter.Thus (3) can be represented as follows: (26)
J
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
trace
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
ψ
T
X
L
X
T
ψ
,where D is diagonal matrix, whose elements are the sum of the column or row elements in symmetric matrix S, represented as D
i
j
=
∑
j
S
i
j, and L is Laplace operator; L
=
D
-
S. It can be seen that LPP is also a particular case of factor analysis framework. The style factor is the weighted accumulation of sample variance in the same neighborhood.


#### 2.3.4. The Relationship with PCA

PCA is a commonly used algorithm for dimensionality reduction in pattern recognition, the objective of which is to find the best representative mapping space of the original data in the minimum mean square sense. The objective function to the datasetX
=
x
i
i
=
1
n
∈
R
m is(27)
J
ψ
=
arg
⁡
min
ψ
T
ψ
=
1
⁡
ψ
Σ
ψ
,where Σ is the covariance matrix:(28)
Σ
=
∑
i
=
1
n
x
i
-
μ
x
i
-
μ
T
=
∑
i
=
1
n
x
i
-
1
n
∑
j
=
1
n
x
j
x
i
-
1
n
∑
k
=
1
n
x
k
T
=
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
i
-
x
j
x
i
-
x
k
T
,where μ is the mean value of all the samples; reformulate Σ in different index, and then(29)
Σ
=
1
n
2
∑
j
=
1
n
∑
i
=
1
n
∑
k
=
1
n
x
j
-
x
i
x
j
-
x
k
T
.Change the summation order in (29); then(30)
Σ
=
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
j
-
x
i
x
j
-
x
k
T
.Add (28) to (30):(31)
Σ
=
1
2
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
i
-
x
j
x
i
-
x
j
T
=
1
2
n
∑
i
=
1
n
∑
j
=
1
n
x
i
-
x
j
x
i
-
x
j
T
.Set W
i
j
s to be an n
×
n matrix in which each element is equal to 1
/
2
n; then style variance matrix F
s can be represented as follows:(32)
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
2
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
1
2
n
x
j
i
-
x
k
i
2
.Treat each sample as an independent partition; then(33)
F
s
=
1
2
n
∑
i
=
1
n
∑
j
=
1
n
x
i
-
x
j
x
i
-
x
j
T
=
Σ
.Thus (3) can be represented as follows:(34)
J
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
t
r
a
c
e
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
ψ
=
1
⁡
ψ
T
Σ
ψ
.It can be seen that PCA is also a particular case of factor analysis framework. The style factor of PCA is the accumulation of all the sample variance.


### 2.4. Related Works and Discussions

Feature extraction methods based on subspace have got a lot of attention. Unifying all the current subspace analysis methods is convenient for comprehending the essence of these methods and comparing the variance of different methods. Various frameworks have been proposed in order to deeply mine the commonality and individuality of current feature extraction methods and provide a novel platform for proposing new methods. In 2007, Yan et al. have proposed the Graph Embedding (GE) framework [20], which describes most current subspace methods and presents that the variance of linear subspace methods is expressed in the graph structure difference, and the variance between linear methods and kernel methods is expressed in different way to embed. In 2009, [21] proposed a Patch Alignment (PA) framework for the motivation of dimensionality reduction. In the same year a universal learning framework of supervised subspace [22] has been proposed, which is based on the fundamental concept of discriminant analysis and evaluates the classification capacity of the features through the ratio between the intraclass compactness and interclass separability after projecting. It proves that LDA, LPP, and NPE algorithms are all particular cases of that framework, and the essence of these algorithms, no matter whether supervised or unsupervised, is to minimize the intraclass compactness while maximizing the interclass separability, only different in the process of sample classes.We propose the framework based on factor analysis through deep research on subspace methods and current unified framework. Factor analysis framework is superior to other frameworks in the following aspects:(1)
The classification meaning is clearer and the design is more flexible. The content and style factors in factor analysis are various depending on the different applications, so kinds of design schemes of the two factors can be proposed by researchers. The intraclass compactness and interclass separability in the universal learning framework of supervised subspace proposed in [22] are exactly the content and style factors in factor analysis framework. The part optimization design in [21] is the process of factor separation. Meanwhile, factor analysis criterion provides flexible weights allocation plan to research.
(2)
Factor analysis framework is adapted to feature extraction and dimensionality reduction methods which are commonly used currently, including linear, nonlinear, and tensor decomposition methods. Supervised and unsupervised feature extraction methods are also included.
(3)
The factor analysis framework is designed for classification, so superior feature extraction methods can be designed through this platform. The FA-LDA algorithm presented in Section3.1 and FA-LPP algorithm presented in Section 3.2 are both improved feature extraction methods proposed using factor analysis framework.


## 2.1. Factors Analysis

Content and style factors can be seen as two independent elements which influence the object and determine the observation [15], as mentioned before. The information contributed to recognition is defined as content factor and the negative information is style factor during pattern recognition.For a general classification problem, the sample set is represented asX
=
[
x
1
,
x
2
,
…
,
x
N
], x
i
∈
R
m, where N denotes the sample number and m denotes the feature dimension. In practice, the feature dimension m is very high usually, including redundancy and interference information. In order to alleviate the curse of dimensionality, it is necessary to transform the original data X from a high-dimensional space to a low-dimensional space which is represented as Y
=
[
y
1
,
y
2
,
…
,
y
N
]
,
y
i
∈
R
m
′. The essence of dimensionality reduction is to find a mapping function F
:
x
→
y
^ that transforms x
∈
R
m into appropriate low-dimensional space y
^
∈
R
m
′, where typically m
≫
m
′:(1)
y
^
=
F
x
.According to the various cases, the mapping function can be explicit or implicit and linear or nonlinear. A dimensionality reduction algorithm based on factor analysis is proposed in this paper, which is different from traditional methods. The objective of the dimensionality reduction algorithm based on factor analysis is to find a low-dimensional datasetY from the original dataset X, in which the style factor is suppressed as much as possible. The objective function is represented as follows:(2)
y
∗
=
arg
⁡
max
⁡
F
c
′
F
s
′
=
arg
⁡
max
⁡
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
μ
i
′
-
μ
j
′
2
W
i
j
c
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
y
j
i
-
y
k
i
2
W
j
k
s
=
arg
⁡
min
F
c
′
=
d
⁡
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
y
j
i
-
y
k
i
2
W
j
k
s
,where F
c
′, F
s
′ are the content and style variance matrices after mapping (factors separating), respectively, C is the total number of partitions, N
i is the sample number of the i
t
h partition, p
i is the prior probability of the i
t
h partition (including the i
t
h class for supervised methods and the i
t
h local manifold space for unsupervised methods, generally named as the i
t
h partition), μ
i
′ is the mean vector of the i
t
h partition after factors separating, y
j
(
i
), y
k
(
i
) are the j
t
h and k
t
h sample vectors of the i
t
h partition, respectively, after factors separating, W
i
j
c, W
j
k
s are weight matrices of content variance and style variance, respectively, and d is the constraint constant, which is different in different cases. Factor analysis criteria make the content variance of the objects between partitions as high as possible and the style variance of the objects within a partition as low as possible. In order to eliminate the style factor of the objects maximally, the style factor should be minimized and the content factor should be maximized after mapping transformation.


## 2.2. General Framework for Feature Extract

There are two vital steps for feature extraction based on factor analysis framework; one is the design of factor separating objective function, mainly including the design of partition and weight matrix, and the other is the design of space mapping function. Partition is designed based on the class information with supervised methods or local neighborhood through unsupervised or semisupervised methods of the samples. Weight matrix is designed according to the classification performance for purpose of finding the optimal distribution scheme, which is biased to the weakly separated samples in the case without sacrificing the strongly separated samples. Space mapping function is designed to transform the samples from input space to feature space. Thus, the research of factor analysis framework can proceed from the following two aspects: the first is to propose different space mapping transformation aimed at different applications, and the second is to find the design scheme of factor separating objective function that matches the pattern recognition best.


### 2.2.1. Space Mapping Transformation

There are three categories of space mapping transformation algorithms, which are linear space, kernel space, and tensor space mapping transformation, respectively.(a) Linear Space Mapping Transformation. The relationship of datasets X and Y is supposed to be linear in linear space mapping transformation; namely, y
i
=
ψ
T
x
i. Thus the objective function (see (2)) is converted to(3)
J
ψ
=
arg
⁡
max
⁡
trace
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
ψ
T
μ
i
0
-
μ
j
0
W
i
j
c
μ
i
0
-
μ
j
0
T
ψ
trace
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
ψ
T
x
j
i
-
x
k
i
W
j
k
s
x
j
i
-
x
k
i
T
ψ
=
arg
⁡
max
⁡
trace
ψ
T
F
c
ψ
trace
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
trace
ψ
T
F
s
ψ
,where μ
i
0 is the mean vector of the i
t
h partition in original space, x
j
(
i
), x
k
(
i
) are the j
t
h and k
t
h sample vectors of the i
t
h partition samples in original space, and F
c, F
s are the content and style variance matrices in original space, respectively, defined as follows:(4)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
W
i
j
c
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
.(b) Kernel Space Mapping Transformation. The direct way to extend methods from linear projections to nonlinear cases is to utilize the kernel trick. The intuition of the kernel methods is to map the data from the original input space to a higher-dimensional Hilbert space (kernel space) as ϕ
:
x
→
H. The content and style variance matrices F
c, F
s in kernel space H are defined as follows:(5)
F
c
ϕ
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
W
i
j
c
μ
i
0
ϕ
-
μ
j
0
ϕ
μ
i
0
ϕ
-
μ
j
0
ϕ
T
,
F
s
ϕ
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
ϕ
x
j
i
-
ϕ
x
k
i
ϕ
x
j
i
-
ϕ
x
k
i
T
,where μ
i
0
ϕ
=
1
/
N
i
∑
i
=
1
N
i
ϕ
(
x
i
0
) is the mean vector of the i
t
h partition samples in kernel space H and ϕ
(
x
j
(
i
)
) is the kernel mapping to x
j
(
i
).(c) Tensor Space Mapping Transformation. Tensorization is another extension of linear methods, and it is a multiple linear extension. Firstly some terminologies [16] on tensor operations are reviewed before introducing the tensor space mapping transformation.Tensor is the extension of vector and matrix. TensorA
∈
R
I
1
×
I
2
×
⋯
×
I
r represents m
r components in m-dimension space, where every component is a function of coordinates, linearly changing with the coordinates following certain rules. r is the order of the tensor. When r
=
1
A is simplified to a vector and when r
=
2
A is simplified to a matrix. Expanding the tensors can facilitate its matrix representation. The k-order expansion of tensor A is A
(
k
)
∈
R
I
k
×
(
I
k
+
1
⋯
I
r
I
1
I
2
⋯
I
r
-
1
). The k-mode product between a tensor A and a matrix U
∈
R
J
k
×
I
k is defined as B
=
A
×
k
U, or B
(
k
)
=
U
A
(
k
) for the form of tensor expansion.Reform the original high-dimensional samples as the tensor{
X
i
∈
R
I
1
×
I
2
×
⋯
×
I
r
,
i
=
1,2
,
…
,
N
}. Corresponding low-dimensional tensors as (6) are obtained from high-dimensional tensors through projecting (multiple linear projection), which is similar to linear methods:(6)
y
i
=
X
i
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
.Then the objective function (see (2)) is converted to (7)
ψ
1
,
…
,
ψ
r
∗
=
arg
⁡
max
⁡
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
μ
i
0
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
-
μ
j
0
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
2
W
i
j
c
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
X
j
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
-
X
k
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
2
W
j
k
s
.


### 2.2.2. The Design of Factor Separating Objective Function

The design of factor separating objective function is mainly about separating the content and style factors of objects, including the design of partitions and weight matrix.There are two schemes for partition design recently; one is based on supervised algorithms, such as LDA, in which the samples are partitioned according to the class information of samples; the other is based on unsupervised algorithms, such as LPP, in which the samples are partitioned according to thek-nearest neighborhood of samples.Variant weighting schemes have been proposed for different cases, which can be classified into four categories according to the weighting algorithms for scatter matrix between classes (content variance matrixF
c) in supervised face recognition methods based on weighted Linear Discriminant Analysis [17].(1) Euclidean Distance Algorithm [17]. Consider(8)
w
i
,
j
=
1
e
d
2
,where e
d is the Euclidean distance between the mean vectors of two classes.(2) Estimating Bayesian Error Rate Algorithm [18]. Consider(9)
w
i
,
j
=
1
2
Δ
i
j
2
erf
⁡
Δ
i
j
2
2
,where Δ
i
j is the Mahalanobis distance between the mean vectors of two classes and erf
⁡
(
x
)
=
2
/
π
∫
0
x
e
-
t
2
d
t is the error function.(3) Confusing Information Algorithm [19]. Consider(10)
w
i
,
j
=
E
i
j
n
i
if
i
≠
j
0
if
i
=
j
,where E
i
j is the number of samples which are misclassified into class j but belong to class i actually and n
i is the sample number of the i
t
h class.(4) Grey Correlation Analysis Algorithm Proposed in This Paper. Consider(11)
w
i
,
j
=
1
n
∑
k
=
1
n
γ
μ
i
k
,
μ
j
k
,where n is the vector dimensionality and μ
i
(
k
), μ
j
(
k
) are the k
t
h feature in the mean vectors of the i
t
h and j
t
h partitions. γ
(
μ
i
(
k
), μ
j
(
k
)
) is the correlation coefficients of the k
t
h feature in the mean vectors of the i
t
h and j
t
h partitions, defined as follows: (12)
γ
μ
i
k
,
μ
j
k
=
min
j
⁡
min
k
⁡
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
.The weighting scheme about the style variance matrixF
s gets little attention recently. According to the weighting scheme in LPP method, a new weighting scheme for F
s based on thermonuclear similarity is proposed in this paper:(13)
W
i
j
=
exp
⁡
x
i
-
x
j
2
2
t
x
i
,
x
j
belong  to  same  partition
0
otherwise.It deserves deep research on how to distribute the weights effectively and reasonably. The optimal weighting scheme will never abandon any object that may be classified correctly and never waste any weight value. In other words, it will help all the objects which need help as much as possible in order to achieve a win-win situation.


## 2.2.1. Space Mapping Transformation

There are three categories of space mapping transformation algorithms, which are linear space, kernel space, and tensor space mapping transformation, respectively.(a) Linear Space Mapping Transformation. The relationship of datasets X and Y is supposed to be linear in linear space mapping transformation; namely, y
i
=
ψ
T
x
i. Thus the objective function (see (2)) is converted to(3)
J
ψ
=
arg
⁡
max
⁡
trace
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
ψ
T
μ
i
0
-
μ
j
0
W
i
j
c
μ
i
0
-
μ
j
0
T
ψ
trace
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
ψ
T
x
j
i
-
x
k
i
W
j
k
s
x
j
i
-
x
k
i
T
ψ
=
arg
⁡
max
⁡
trace
ψ
T
F
c
ψ
trace
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
trace
ψ
T
F
s
ψ
,where μ
i
0 is the mean vector of the i
t
h partition in original space, x
j
(
i
), x
k
(
i
) are the j
t
h and k
t
h sample vectors of the i
t
h partition samples in original space, and F
c, F
s are the content and style variance matrices in original space, respectively, defined as follows:(4)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
W
i
j
c
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
.(b) Kernel Space Mapping Transformation. The direct way to extend methods from linear projections to nonlinear cases is to utilize the kernel trick. The intuition of the kernel methods is to map the data from the original input space to a higher-dimensional Hilbert space (kernel space) as ϕ
:
x
→
H. The content and style variance matrices F
c, F
s in kernel space H are defined as follows:(5)
F
c
ϕ
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
W
i
j
c
μ
i
0
ϕ
-
μ
j
0
ϕ
μ
i
0
ϕ
-
μ
j
0
ϕ
T
,
F
s
ϕ
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
ϕ
x
j
i
-
ϕ
x
k
i
ϕ
x
j
i
-
ϕ
x
k
i
T
,where μ
i
0
ϕ
=
1
/
N
i
∑
i
=
1
N
i
ϕ
(
x
i
0
) is the mean vector of the i
t
h partition samples in kernel space H and ϕ
(
x
j
(
i
)
) is the kernel mapping to x
j
(
i
).(c) Tensor Space Mapping Transformation. Tensorization is another extension of linear methods, and it is a multiple linear extension. Firstly some terminologies [16] on tensor operations are reviewed before introducing the tensor space mapping transformation.Tensor is the extension of vector and matrix. TensorA
∈
R
I
1
×
I
2
×
⋯
×
I
r represents m
r components in m-dimension space, where every component is a function of coordinates, linearly changing with the coordinates following certain rules. r is the order of the tensor. When r
=
1
A is simplified to a vector and when r
=
2
A is simplified to a matrix. Expanding the tensors can facilitate its matrix representation. The k-order expansion of tensor A is A
(
k
)
∈
R
I
k
×
(
I
k
+
1
⋯
I
r
I
1
I
2
⋯
I
r
-
1
). The k-mode product between a tensor A and a matrix U
∈
R
J
k
×
I
k is defined as B
=
A
×
k
U, or B
(
k
)
=
U
A
(
k
) for the form of tensor expansion.Reform the original high-dimensional samples as the tensor{
X
i
∈
R
I
1
×
I
2
×
⋯
×
I
r
,
i
=
1,2
,
…
,
N
}. Corresponding low-dimensional tensors as (6) are obtained from high-dimensional tensors through projecting (multiple linear projection), which is similar to linear methods:(6)
y
i
=
X
i
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
.Then the objective function (see (2)) is converted to (7)
ψ
1
,
…
,
ψ
r
∗
=
arg
⁡
max
⁡
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
μ
i
0
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
-
μ
j
0
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
2
W
i
j
c
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
X
j
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
-
X
k
×
1
ψ
1
×
2
ψ
2
⋯
×
r
ψ
r
2
W
j
k
s
.


## 2.2.2. The Design of Factor Separating Objective Function

The design of factor separating objective function is mainly about separating the content and style factors of objects, including the design of partitions and weight matrix.There are two schemes for partition design recently; one is based on supervised algorithms, such as LDA, in which the samples are partitioned according to the class information of samples; the other is based on unsupervised algorithms, such as LPP, in which the samples are partitioned according to thek-nearest neighborhood of samples.Variant weighting schemes have been proposed for different cases, which can be classified into four categories according to the weighting algorithms for scatter matrix between classes (content variance matrixF
c) in supervised face recognition methods based on weighted Linear Discriminant Analysis [17].(1) Euclidean Distance Algorithm [17]. Consider(8)
w
i
,
j
=
1
e
d
2
,where e
d is the Euclidean distance between the mean vectors of two classes.(2) Estimating Bayesian Error Rate Algorithm [18]. Consider(9)
w
i
,
j
=
1
2
Δ
i
j
2
erf
⁡
Δ
i
j
2
2
,where Δ
i
j is the Mahalanobis distance between the mean vectors of two classes and erf
⁡
(
x
)
=
2
/
π
∫
0
x
e
-
t
2
d
t is the error function.(3) Confusing Information Algorithm [19]. Consider(10)
w
i
,
j
=
E
i
j
n
i
if
i
≠
j
0
if
i
=
j
,where E
i
j is the number of samples which are misclassified into class j but belong to class i actually and n
i is the sample number of the i
t
h class.(4) Grey Correlation Analysis Algorithm Proposed in This Paper. Consider(11)
w
i
,
j
=
1
n
∑
k
=
1
n
γ
μ
i
k
,
μ
j
k
,where n is the vector dimensionality and μ
i
(
k
), μ
j
(
k
) are the k
t
h feature in the mean vectors of the i
t
h and j
t
h partitions. γ
(
μ
i
(
k
), μ
j
(
k
)
) is the correlation coefficients of the k
t
h feature in the mean vectors of the i
t
h and j
t
h partitions, defined as follows: (12)
γ
μ
i
k
,
μ
j
k
=
min
j
⁡
min
k
⁡
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
.The weighting scheme about the style variance matrixF
s gets little attention recently. According to the weighting scheme in LPP method, a new weighting scheme for F
s based on thermonuclear similarity is proposed in this paper:(13)
W
i
j
=
exp
⁡
x
i
-
x
j
2
2
t
x
i
,
x
j
belong  to  same  partition
0
otherwise.It deserves deep research on how to distribute the weights effectively and reasonably. The optimal weighting scheme will never abandon any object that may be classified correctly and never waste any weight value. In other words, it will help all the objects which need help as much as possible in order to achieve a win-win situation.


## 2.3. Unifying Various Algorithms of Feature Extraction

The relationship between several classical subspace dimensionality reduction algorithms and factor analysis framework is discussed in this section. Actually, most subspace algorithms that are commonly used recently can be represented and explained by factor analysis framework. After unifying current subspace algorithms in framework, it is obvious that these algorithms are just different in the basis of factor construction and constraint of factor separation, though the motivation or objective of these algorithms is varying. Next we will introduce the relationships between factor analysis and some typical methods of dimension reduction based on subspace.


### 2.3.1. The Relationship with LDA

The objective of LDA is to find the optimal projection direction of classification that maximizes the ratio between the intraclass and interclass scatter matrix:(14)
J
ψ
=
arg
⁡
max
⁡
trace
ψ
T
S
b
ψ
trace
ψ
T
S
w
ψ
,
(15)
S
b
=
∑
i
=
1
C
p
i
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
,
(16)
S
w
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
x
j
0
-
μ
i
0
x
j
0
-
μ
i
0
T
,where C is the class number, p
i is the prior probability of class i, μ
i
0 is the mean vector of class i before projection, μ
0 is the overall mean vector before projection, and N
i is the sample number of class i. Refer to (3) and (4), set the weight matrix of F
c and F
s as all 1 matrix, and then(17)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
.After further conversion,(18)
F
c
=
∑
i
=
1
C
p
i
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
=
S
b
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
x
j
0
-
μ
i
0
x
j
0
-
μ
i
0
T
=
S
w
.Substitute (17) into (14); then(19)
J
ψ
=
arg
⁡
max
⁡
trace
ψ
T
S
b
ψ
trace
ψ
T
S
w
ψ
=
trace
ψ
T
F
c
ψ
trace
ψ
T
F
s
ψ
.Thus, LDA is a particular case of factor analysis framework when the entire weight matrix is 1 and the basis of partition is class information. The content factor is the accumulation of mean variance among each class and the style factor is the variance of each sample in the same class.


### 2.3.2. The Relationship with MFA [20] and DLA [21]

In order to overcome the hypothesis of LDA that the samples should be of Gaussian distribution, MFA redefines the scatter matrix within class and between classes of LDA, and the objective function is as follows:(20)
J
ψ
=
arg
⁡
max
⁡
∑
c
∑
i
,
j
∈
P
k
2
c
or
j
,
i
∈
P
k
2
c
′
ψ
T
x
i
c
-
ψ
T
x
j
c
′
2
∑
c
=
1
C
∑
j
∈
N
K
1
c
i
or
i
∈
N
k
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
,where c is the class, P
k
2
(
c
) is a data pairs set that are the k
2-nearest pairs between two different classes c and c
′, and N
k
1
c
(
i
) indicates the index set of the k
1-nearest neighbors of the sample i within the c
t
h class.It can be concluded that MFA is also a particular case of factor analysis framework; only it considers both the class information and nearest neighborhood of the samples for the partition design. The numerator of (20) can be rewritten as follows:(21)
∑
c
∑
i
,
j
∈
P
k
2
c
or
j
,
i
∈
P
k
2
c
′
ψ
T
x
i
c
-
ψ
T
x
j
c
′
2
=
∑
i
=
1
C
-
1
∑
j
=
1
C
ψ
T
μ
i
0
-
ψ
T
μ
j
0
2
=
ψ
T
F
c
ψ
,where μ
i
0, μ
j
0 are the mean vector of the k
2-nearest pairs between the i
t
h and j
t
h classes. Therefore, the interclass separability can be seen as the content factor in factor analysis framework. Though the class information and nearest neighborhood are considered in the design of interclass separability, the weight matrix is ignored.Similarly, the denominator of (20) can be rewritten as follows:(22)
∑
c
=
1
C
∑
j
∈
N
K
1
C
i
or
i
∈
N
k
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
=
∑
c
=
1
C
∑
i
=
1
N
K
1
c
i
∑
j
=
1
N
K
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
=
ψ
T
F
s
ψ
.Therefore, the intraclass compactness of MFA can be seen as the style factor in factor analysis framework. Interclass separability is conducted based on thek-nearest neighbor relationships of the samples from the same class, disregarding the design of weight matrix.For DLA, the design criterion of part optimization is that the distance of the nearest neighbor image is minimized within class and maximized among classes. The objective function of DLA is equal to (20), and thus DLA is a particular case of factor analysis framework also.


### 2.3.3. The Relationship with LPP

Refer to (3); J
(
ψ
)
=
arg
⁡
m
i
n
ψ
T
F
c
ψ
=
d
trace
ψ
T
F
s
ψ. Assume that all the prior probability p
i is identical, and then the content and style variance matrix can be represented as follows:(23)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
W
i
j
c
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
=
∑
i
=
1
C
W
i
j
c
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
,
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
,where F
s can be represented as follows:(24)
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
2
.Partition the samples according to k-nearest neighbors; then (24) can be written as(25)
F
s
=
∑
i
N
∑
j
N
x
i
-
x
j
2
w
i
j
,where W
∈
R
N
×
N is the relation matrix weighted by the heat kernels: w
(
i
,
j
)
=
exp
⁡
⁡
(
-
x
i
-
x
j
2
/
t
) if x
i is one of the k-nearest neighbors of x
j or x
j is one of the k-nearest neighbors of x
i; otherwise it is 0, and t is a tuning parameter.Thus (3) can be represented as follows: (26)
J
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
trace
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
ψ
T
X
L
X
T
ψ
,where D is diagonal matrix, whose elements are the sum of the column or row elements in symmetric matrix S, represented as D
i
j
=
∑
j
S
i
j, and L is Laplace operator; L
=
D
-
S. It can be seen that LPP is also a particular case of factor analysis framework. The style factor is the weighted accumulation of sample variance in the same neighborhood.


### 2.3.4. The Relationship with PCA

PCA is a commonly used algorithm for dimensionality reduction in pattern recognition, the objective of which is to find the best representative mapping space of the original data in the minimum mean square sense. The objective function to the datasetX
=
x
i
i
=
1
n
∈
R
m is(27)
J
ψ
=
arg
⁡
min
ψ
T
ψ
=
1
⁡
ψ
Σ
ψ
,where Σ is the covariance matrix:(28)
Σ
=
∑
i
=
1
n
x
i
-
μ
x
i
-
μ
T
=
∑
i
=
1
n
x
i
-
1
n
∑
j
=
1
n
x
j
x
i
-
1
n
∑
k
=
1
n
x
k
T
=
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
i
-
x
j
x
i
-
x
k
T
,where μ is the mean value of all the samples; reformulate Σ in different index, and then(29)
Σ
=
1
n
2
∑
j
=
1
n
∑
i
=
1
n
∑
k
=
1
n
x
j
-
x
i
x
j
-
x
k
T
.Change the summation order in (29); then(30)
Σ
=
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
j
-
x
i
x
j
-
x
k
T
.Add (28) to (30):(31)
Σ
=
1
2
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
i
-
x
j
x
i
-
x
j
T
=
1
2
n
∑
i
=
1
n
∑
j
=
1
n
x
i
-
x
j
x
i
-
x
j
T
.Set W
i
j
s to be an n
×
n matrix in which each element is equal to 1
/
2
n; then style variance matrix F
s can be represented as follows:(32)
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
2
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
1
2
n
x
j
i
-
x
k
i
2
.Treat each sample as an independent partition; then(33)
F
s
=
1
2
n
∑
i
=
1
n
∑
j
=
1
n
x
i
-
x
j
x
i
-
x
j
T
=
Σ
.Thus (3) can be represented as follows:(34)
J
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
t
r
a
c
e
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
ψ
=
1
⁡
ψ
T
Σ
ψ
.It can be seen that PCA is also a particular case of factor analysis framework. The style factor of PCA is the accumulation of all the sample variance.


## 2.3.1. The Relationship with LDA

The objective of LDA is to find the optimal projection direction of classification that maximizes the ratio between the intraclass and interclass scatter matrix:(14)
J
ψ
=
arg
⁡
max
⁡
trace
ψ
T
S
b
ψ
trace
ψ
T
S
w
ψ
,
(15)
S
b
=
∑
i
=
1
C
p
i
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
,
(16)
S
w
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
x
j
0
-
μ
i
0
x
j
0
-
μ
i
0
T
,where C is the class number, p
i is the prior probability of class i, μ
i
0 is the mean vector of class i before projection, μ
0 is the overall mean vector before projection, and N
i is the sample number of class i. Refer to (3) and (4), set the weight matrix of F
c and F
s as all 1 matrix, and then(17)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
.After further conversion,(18)
F
c
=
∑
i
=
1
C
p
i
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
=
S
b
,
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
x
j
0
-
μ
i
0
x
j
0
-
μ
i
0
T
=
S
w
.Substitute (17) into (14); then(19)
J
ψ
=
arg
⁡
max
⁡
trace
ψ
T
S
b
ψ
trace
ψ
T
S
w
ψ
=
trace
ψ
T
F
c
ψ
trace
ψ
T
F
s
ψ
.Thus, LDA is a particular case of factor analysis framework when the entire weight matrix is 1 and the basis of partition is class information. The content factor is the accumulation of mean variance among each class and the style factor is the variance of each sample in the same class.


## 2.3.2. The Relationship with MFA [20] and DLA [21]

In order to overcome the hypothesis of LDA that the samples should be of Gaussian distribution, MFA redefines the scatter matrix within class and between classes of LDA, and the objective function is as follows:(20)
J
ψ
=
arg
⁡
max
⁡
∑
c
∑
i
,
j
∈
P
k
2
c
or
j
,
i
∈
P
k
2
c
′
ψ
T
x
i
c
-
ψ
T
x
j
c
′
2
∑
c
=
1
C
∑
j
∈
N
K
1
c
i
or
i
∈
N
k
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
,where c is the class, P
k
2
(
c
) is a data pairs set that are the k
2-nearest pairs between two different classes c and c
′, and N
k
1
c
(
i
) indicates the index set of the k
1-nearest neighbors of the sample i within the c
t
h class.It can be concluded that MFA is also a particular case of factor analysis framework; only it considers both the class information and nearest neighborhood of the samples for the partition design. The numerator of (20) can be rewritten as follows:(21)
∑
c
∑
i
,
j
∈
P
k
2
c
or
j
,
i
∈
P
k
2
c
′
ψ
T
x
i
c
-
ψ
T
x
j
c
′
2
=
∑
i
=
1
C
-
1
∑
j
=
1
C
ψ
T
μ
i
0
-
ψ
T
μ
j
0
2
=
ψ
T
F
c
ψ
,where μ
i
0, μ
j
0 are the mean vector of the k
2-nearest pairs between the i
t
h and j
t
h classes. Therefore, the interclass separability can be seen as the content factor in factor analysis framework. Though the class information and nearest neighborhood are considered in the design of interclass separability, the weight matrix is ignored.Similarly, the denominator of (20) can be rewritten as follows:(22)
∑
c
=
1
C
∑
j
∈
N
K
1
C
i
or
i
∈
N
k
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
=
∑
c
=
1
C
∑
i
=
1
N
K
1
c
i
∑
j
=
1
N
K
1
c
j
ψ
T
x
i
c
-
ψ
T
x
j
c
2
=
ψ
T
F
s
ψ
.Therefore, the intraclass compactness of MFA can be seen as the style factor in factor analysis framework. Interclass separability is conducted based on thek-nearest neighbor relationships of the samples from the same class, disregarding the design of weight matrix.For DLA, the design criterion of part optimization is that the distance of the nearest neighbor image is minimized within class and maximized among classes. The objective function of DLA is equal to (20), and thus DLA is a particular case of factor analysis framework also.


## 2.3.3. The Relationship with LPP

Refer to (3); J
(
ψ
)
=
arg
⁡
m
i
n
ψ
T
F
c
ψ
=
d
trace
ψ
T
F
s
ψ. Assume that all the prior probability p
i is identical, and then the content and style variance matrix can be represented as follows:(23)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
W
i
j
c
μ
i
0
-
μ
j
0
μ
i
0
-
μ
j
0
T
=
∑
i
=
1
C
W
i
j
c
μ
i
0
-
μ
0
μ
i
0
-
μ
0
T
,
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
,where F
s can be represented as follows:(24)
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
2
.Partition the samples according to k-nearest neighbors; then (24) can be written as(25)
F
s
=
∑
i
N
∑
j
N
x
i
-
x
j
2
w
i
j
,where W
∈
R
N
×
N is the relation matrix weighted by the heat kernels: w
(
i
,
j
)
=
exp
⁡
⁡
(
-
x
i
-
x
j
2
/
t
) if x
i is one of the k-nearest neighbors of x
j or x
j is one of the k-nearest neighbors of x
i; otherwise it is 0, and t is a tuning parameter.Thus (3) can be represented as follows: (26)
J
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
trace
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
ψ
T
X
L
X
T
ψ
,where D is diagonal matrix, whose elements are the sum of the column or row elements in symmetric matrix S, represented as D
i
j
=
∑
j
S
i
j, and L is Laplace operator; L
=
D
-
S. It can be seen that LPP is also a particular case of factor analysis framework. The style factor is the weighted accumulation of sample variance in the same neighborhood.


## 2.3.4. The Relationship with PCA

PCA is a commonly used algorithm for dimensionality reduction in pattern recognition, the objective of which is to find the best representative mapping space of the original data in the minimum mean square sense. The objective function to the datasetX
=
x
i
i
=
1
n
∈
R
m is(27)
J
ψ
=
arg
⁡
min
ψ
T
ψ
=
1
⁡
ψ
Σ
ψ
,where Σ is the covariance matrix:(28)
Σ
=
∑
i
=
1
n
x
i
-
μ
x
i
-
μ
T
=
∑
i
=
1
n
x
i
-
1
n
∑
j
=
1
n
x
j
x
i
-
1
n
∑
k
=
1
n
x
k
T
=
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
i
-
x
j
x
i
-
x
k
T
,where μ is the mean value of all the samples; reformulate Σ in different index, and then(29)
Σ
=
1
n
2
∑
j
=
1
n
∑
i
=
1
n
∑
k
=
1
n
x
j
-
x
i
x
j
-
x
k
T
.Change the summation order in (29); then(30)
Σ
=
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
j
-
x
i
x
j
-
x
k
T
.Add (28) to (30):(31)
Σ
=
1
2
1
n
2
∑
i
=
1
n
∑
j
=
1
n
∑
k
=
1
n
x
i
-
x
j
x
i
-
x
j
T
=
1
2
n
∑
i
=
1
n
∑
j
=
1
n
x
i
-
x
j
x
i
-
x
j
T
.Set W
i
j
s to be an n
×
n matrix in which each element is equal to 1
/
2
n; then style variance matrix F
s can be represented as follows:(32)
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
W
j
k
s
x
j
i
-
x
k
i
2
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
1
2
n
x
j
i
-
x
k
i
2
.Treat each sample as an independent partition; then(33)
F
s
=
1
2
n
∑
i
=
1
n
∑
j
=
1
n
x
i
-
x
j
x
i
-
x
j
T
=
Σ
.Thus (3) can be represented as follows:(34)
J
ψ
=
arg
⁡
min
ψ
T
F
c
ψ
=
d
⁡
t
r
a
c
e
ψ
T
F
s
ψ
=
arg
⁡
min
ψ
T
ψ
=
1
⁡
ψ
T
Σ
ψ
.It can be seen that PCA is also a particular case of factor analysis framework. The style factor of PCA is the accumulation of all the sample variance.


## 2.4. Related Works and Discussions

Feature extraction methods based on subspace have got a lot of attention. Unifying all the current subspace analysis methods is convenient for comprehending the essence of these methods and comparing the variance of different methods. Various frameworks have been proposed in order to deeply mine the commonality and individuality of current feature extraction methods and provide a novel platform for proposing new methods. In 2007, Yan et al. have proposed the Graph Embedding (GE) framework [20], which describes most current subspace methods and presents that the variance of linear subspace methods is expressed in the graph structure difference, and the variance between linear methods and kernel methods is expressed in different way to embed. In 2009, [21] proposed a Patch Alignment (PA) framework for the motivation of dimensionality reduction. In the same year a universal learning framework of supervised subspace [22] has been proposed, which is based on the fundamental concept of discriminant analysis and evaluates the classification capacity of the features through the ratio between the intraclass compactness and interclass separability after projecting. It proves that LDA, LPP, and NPE algorithms are all particular cases of that framework, and the essence of these algorithms, no matter whether supervised or unsupervised, is to minimize the intraclass compactness while maximizing the interclass separability, only different in the process of sample classes.We propose the framework based on factor analysis through deep research on subspace methods and current unified framework. Factor analysis framework is superior to other frameworks in the following aspects:(1)
The classification meaning is clearer and the design is more flexible. The content and style factors in factor analysis are various depending on the different applications, so kinds of design schemes of the two factors can be proposed by researchers. The intraclass compactness and interclass separability in the universal learning framework of supervised subspace proposed in [22] are exactly the content and style factors in factor analysis framework. The part optimization design in [21] is the process of factor separation. Meanwhile, factor analysis criterion provides flexible weights allocation plan to research.
(2)
Factor analysis framework is adapted to feature extraction and dimensionality reduction methods which are commonly used currently, including linear, nonlinear, and tensor decomposition methods. Supervised and unsupervised feature extraction methods are also included.
(3)
The factor analysis framework is designed for classification, so superior feature extraction methods can be designed through this platform. The FA-LDA algorithm presented in Section3.1 and FA-LPP algorithm presented in Section 3.2 are both improved feature extraction methods proposed using factor analysis framework.


## 3. FA-LDA and FA-LPP

### 3.1. FA-LDA: Weight Design Based on Factor Analysis

In addition to encompassing most popular dimensionality reduction algorithms, the proposed framework can also be used as a general platform to design new algorithms for dimensionality reduction. With this framework, we develop a new dimensionality reduction algorithm to avoid certain limitations of the traditional Linear Discriminant Analysis in terms of suboptimal problem.Researchers for LDA are mainly focused on the Fisher criterion solution and small sample problems but little on the Fisher criterion itself. Actually, Fisher criterion is suboptimal for the classification problem withK
(
K
>
2
) classes. Marco Loog is one of the earliest researchers that pointed out this problem and proposed an improved weighting algorithm based on Bayesian error rate function [18]. In the meanwhile, Li et al. proposed an improved weighting algorithm based on Euclidean distance at the same year [17]. After that, fractional-step LDA is proposed in [23] to solve the suboptimal problem of Fisher criterion. Fractional-step LDA is effective but time consuming since it is an iterative algorithm. In this section, we improve the Fisher criterion based on factor analysis in order to solve its suboptimal problem.Actually, the content factorF
c and style factor F
s in factor analysis criterion are identical to interclass and intraclass scatter matrices S
b, S
w in the case that every weight equals 1. When there is a pair of classes far from each other, the covariance (
μ
i
-
μ
j
)
(
μ
i
-
μ
j
)
T will be large and totally determine S
b, leading to the fact that the final projecting matrix excessively emphasizes the classes which are already classified well but make less use of the pair of classes (
i
′
,
j
′
) near each other. However, the pairs of classes near each other deserve more attention.According to the above discussion, the weight should be reduced for the class pairs far away from each other while it should be increased for the class pairs near each other.F
c is obtained from weighting S
b:(35)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
w
i
,
j
μ
i
-
μ
j
μ
i
-
μ
j
T
,where(36)
w
i
,
j
=
1
n
∑
k
=
1
n
γ
μ
i
k
,
μ
j
k
,where n is the vector dimensionality, μ
i
(
k
), μ
j
(
k
) are the k
t
h feature in the mean vector of the i
t
h and j
t
h class, respectively, and γ
(
μ
i
(
k
), μ
j
(
k
)
) is the correlation coefficient of the k
t
h feature in the mean vector of the i
t
h and j
t
h class, defined as follows: (37)
γ
μ
i
k
,
μ
j
k
=
min
j
⁡
min
k
⁡
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
.Similarly, the objective of the factor analysis criterion is to remove the style factor as much as possible. Thus, the contribution to solve the eigenvalue from the sample pair near each other in the same class should be reduced, since it is possible that this pair of samples has little interference from style factor; the contribution to solve the eigenvalue from the sample pair far away from each other in the same class should be increased, since it is certain that one sample of this pair contains style factor, and the optimization function of the criterion should emphasize this kind of pairs.F
s is obtained from weighting S
w:(38)
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
w
j
,
k
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
,where the weighting function w
(
j
,
k
) is defined as follows:(39)
w
j
,
k
=
exp
⁡
-
x
j
-
x
k
2
t
2
2
,where t is an empirical constant. The distance within class is more, the weight in S
w
-
1 is heavier, and vice versa.Given the weightedF
c, F
s, FA-LDA is defined as follows:(40)
J
F
W
=
arg
⁡
max
W
⁡
t
r
a
c
e
W
T
F
c
W
t
r
a
c
e
W
T
F
s
W
.


### 3.2. FA-LPP

The style variance in the same locality manifold is minimized after projecting in LPP algorithm; namely, the locality manifold is preserved in low-dimensional space after projecting. We improve the LPP algorithm based on factor analysis criterion so that the style variance in locality manifold of the same class is minimized and the content variance in locality manifold of different classes is maximized after projecting. Thus, the style factor of FA-LPP is(41)
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
w
j
k
s
x
j
i
-
x
k
i
2
,where weight matrix w
j
k
s is(42)
w
j
k
s
=
exp
⁡
x
j
-
x
k
2
2
t
x
j
and
x
k
belong  to  the  same  class  and
x
j
is  one  of  the
k
-nearest  neighbors  of
x
k
0
o
t
h
e
r
.The content factor of FA-LPP is(43)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
w
i
,
j
μ
i
-
μ
j
μ
i
-
μ
j
T
.μ
i, μ
j are the mean values of the k-nearest neighbors of the i
t
h and j
t
h classes, respectively, and w
(
i
,
j
) is computed by (36). Given the definition of F
c, F
s, FA-LPP can be defined as follows:(44)
J
F
W
=
arg
⁡
max
W
⁡
t
r
a
c
e
W
T
F
c
W
t
r
a
c
e
W
T
F
s
W
.


## 3.1. FA-LDA: Weight Design Based on Factor Analysis

In addition to encompassing most popular dimensionality reduction algorithms, the proposed framework can also be used as a general platform to design new algorithms for dimensionality reduction. With this framework, we develop a new dimensionality reduction algorithm to avoid certain limitations of the traditional Linear Discriminant Analysis in terms of suboptimal problem.Researchers for LDA are mainly focused on the Fisher criterion solution and small sample problems but little on the Fisher criterion itself. Actually, Fisher criterion is suboptimal for the classification problem withK
(
K
>
2
) classes. Marco Loog is one of the earliest researchers that pointed out this problem and proposed an improved weighting algorithm based on Bayesian error rate function [18]. In the meanwhile, Li et al. proposed an improved weighting algorithm based on Euclidean distance at the same year [17]. After that, fractional-step LDA is proposed in [23] to solve the suboptimal problem of Fisher criterion. Fractional-step LDA is effective but time consuming since it is an iterative algorithm. In this section, we improve the Fisher criterion based on factor analysis in order to solve its suboptimal problem.Actually, the content factorF
c and style factor F
s in factor analysis criterion are identical to interclass and intraclass scatter matrices S
b, S
w in the case that every weight equals 1. When there is a pair of classes far from each other, the covariance (
μ
i
-
μ
j
)
(
μ
i
-
μ
j
)
T will be large and totally determine S
b, leading to the fact that the final projecting matrix excessively emphasizes the classes which are already classified well but make less use of the pair of classes (
i
′
,
j
′
) near each other. However, the pairs of classes near each other deserve more attention.According to the above discussion, the weight should be reduced for the class pairs far away from each other while it should be increased for the class pairs near each other.F
c is obtained from weighting S
b:(35)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
p
i
p
j
w
i
,
j
μ
i
-
μ
j
μ
i
-
μ
j
T
,where(36)
w
i
,
j
=
1
n
∑
k
=
1
n
γ
μ
i
k
,
μ
j
k
,where n is the vector dimensionality, μ
i
(
k
), μ
j
(
k
) are the k
t
h feature in the mean vector of the i
t
h and j
t
h class, respectively, and γ
(
μ
i
(
k
), μ
j
(
k
)
) is the correlation coefficient of the k
t
h feature in the mean vector of the i
t
h and j
t
h class, defined as follows: (37)
γ
μ
i
k
,
μ
j
k
=
min
j
⁡
min
k
⁡
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
μ
i
k
-
μ
j
k
+
ρ
max
j
⁡
max
k
⁡
μ
i
k
-
μ
j
k
.Similarly, the objective of the factor analysis criterion is to remove the style factor as much as possible. Thus, the contribution to solve the eigenvalue from the sample pair near each other in the same class should be reduced, since it is possible that this pair of samples has little interference from style factor; the contribution to solve the eigenvalue from the sample pair far away from each other in the same class should be increased, since it is certain that one sample of this pair contains style factor, and the optimization function of the criterion should emphasize this kind of pairs.F
s is obtained from weighting S
w:(38)
F
s
=
∑
i
=
1
C
p
i
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
w
j
,
k
x
j
i
-
x
k
i
x
j
i
-
x
k
i
T
,where the weighting function w
(
j
,
k
) is defined as follows:(39)
w
j
,
k
=
exp
⁡
-
x
j
-
x
k
2
t
2
2
,where t is an empirical constant. The distance within class is more, the weight in S
w
-
1 is heavier, and vice versa.Given the weightedF
c, F
s, FA-LDA is defined as follows:(40)
J
F
W
=
arg
⁡
max
W
⁡
t
r
a
c
e
W
T
F
c
W
t
r
a
c
e
W
T
F
s
W
.


## 3.2. FA-LPP

The style variance in the same locality manifold is minimized after projecting in LPP algorithm; namely, the locality manifold is preserved in low-dimensional space after projecting. We improve the LPP algorithm based on factor analysis criterion so that the style variance in locality manifold of the same class is minimized and the content variance in locality manifold of different classes is maximized after projecting. Thus, the style factor of FA-LPP is(41)
F
s
=
∑
i
=
1
C
∑
j
=
1
N
i
-
1
∑
k
=
j
+
1
N
i
w
j
k
s
x
j
i
-
x
k
i
2
,where weight matrix w
j
k
s is(42)
w
j
k
s
=
exp
⁡
x
j
-
x
k
2
2
t
x
j
and
x
k
belong  to  the  same  class  and
x
j
is  one  of  the
k
-nearest  neighbors  of
x
k
0
o
t
h
e
r
.The content factor of FA-LPP is(43)
F
c
=
∑
i
=
1
C
-
1
∑
j
=
i
+
1
C
w
i
,
j
μ
i
-
μ
j
μ
i
-
μ
j
T
.μ
i, μ
j are the mean values of the k-nearest neighbors of the i
t
h and j
t
h classes, respectively, and w
(
i
,
j
) is computed by (36). Given the definition of F
c, F
s, FA-LPP can be defined as follows:(44)
J
F
W
=
arg
⁡
max
W
⁡
t
r
a
c
e
W
T
F
c
W
t
r
a
c
e
W
T
F
s
W
.


## 4. Experiments

### 4.1. FA-LDA

To evaluate the proposed FA-LDA algorithm, we compare it with the LDA algorithm on artificial data and real-world face dataset. During the testing phases, the nearest neighbor (NN) rule was used in classification. Note that [24] proposed a probabilistic graphical model framework to mimic the process of generating a face image. However, they focus on addressing the obstacles of small sample set, occlusion, and illumination variations in one sample face identification. So we do not compare our algorithm with [24].Experiments on Artificial Data. The artificial data includes 3 subsets with 3, 4, and 5 classes, respectively, and the prior probability of each class is identical. Each class contains 60 samples with 9 dimensions. There is one class easy to discriminate in each subset. Thus, its within class covariance matrix plays a dominant role compared to other classes. Meanwhile, assume that the other classes obey the normal distribution and have identical within class covariance matrix. In experiments, 10 samples are selected as train set and the other 50 samples are equally divided into 10 groups as test set. The experiments of each class are conducted 10 times and each experiment is repeated 10 times. The classification rate on average and variance of the two algorithms are listed in Table 1. It can be seen that the classification performance of LDA algorithm is not good on this artificial data, which indicates the limitation of LDA. FA-LDA is superior to LDA in classification property. Since LDA algorithm simply uses the average of the within class scatter matrix as unified covariance matrix, it leads to an overemphasis on the scatter matrix of the class easy to discriminate. However, for FA-LDA algorithm the weights of this easily separated class are small and more attention is on the confusable classes.Table 1
Classification accuracies of LDA and FA-LDA on the artificial database.
3 classes
4 classes
5 classes
FA-LDA
LDA
FA-LDA
LDA
FA-LDA
LDA
83.5 ± 7.45
62 ± 8.34
75.5 ± 5.6
38.54 ± 10.21
78.26 ± 5.4
45.26 ± 7.85
81 ± 9.01
56.7 ± 7.45
73 ± 6.68
45.85 ± 7.37
75.66 ± 6.28
46.15 ± 8.21
83.5 ± 9.07
59.5 ± 10.53
73.5 ± 5.62
46.33 ± 5.36
75.5 ± 7.25
39.54 ± 9.25
80.5 ± 8.67
57 ± 9.57
76.67 ± 5.6
42.64 ± 7.6
79.45 ± 5.5
48.26 ± 6.51
81 ± 9.01
53.5 ± 12.45
78.56 ± 5.21
43.85 ± 7.49
76.5 ± 7.50
44.58 ± 9.21
81.5 ± 6.26
61.5 ± 9.57
78.85 ± 5.34
40.28 ± 6.35
70.57 ± 9.14
39.83 ± 5.68
84.5 ± 6.58
60 ± 6.7
80.21 ± 8.63
42.84 ± 5.49
75.64 ± 5.60
46.25 ± 7.35
82.5 ± 8.08
59.5 ± 11.56
78.26 ± 5.23
41.45 ± 8.16
76.42 ± 5.60
42.35 ± 5.36
81 ± 8.45
64.5 ± 6.87
75.95 ± 6.72
43.58 ± 6.53
77.38 ± 6.58
46.25 ± 8.24Experiments on Real-World Data. The ORL database contains 400 images of 40 individuals which is used for real-world experiment. The images were captured at different times and with different variations including expression and facial details, which can be seen from Figure 2. In order to evaluate the superiority of FA-LDA, we compare it with the LDA and current weighted algorithms, in which WLDA1 [17] is based on Euclidean distance and WLDA2 [19] is based on confusion matrix. For training, we randomly selected different numbers (three, four, five, six, seven, eight, and nine) of images per individual and used the rest of the images for testing. Such a trial was independently performed 10 times, and then the average recognition results were calculated. The results are shown in Figure 3. The classification property of LDA is the worst, which illustrated the validity the weighting scheme based on factor analysis criterion. FA-LDA algorithm achieves the best performance for it not only weights S
b to reduce the influence of the far away class, but also weights S
w to improve the anti-interference ability of discriminant criterion.Figure 2
The sample facial images of ORL database.Figure 3
Best recognition rates of four algorithms on the testing sets of ORL.


### 4.2. FA-LPP

To evaluate the proposed FA-LPP algorithm, we compare it with LPP and MFA/DLA on ORL and Yale database. The Yale database is built by computational vision and control center at Yale University. Everyone has 11 images varying in expressions (blinking, glad, sad, or surprised), illumination conditions (frontal, left, or right), and facial details (glasses or no glasses), which can be seen from Figure4.Figure 4
The sample facial images of Yale database.For the ORL and Yale database, the image set is partitioned into the different gallery and probe sets whereG
m
/
P
n indicates that m images per person are randomly selected for training and the remaining n images are used for testing. For the LPP, the important parameters include k (the number of Neighbor Measurements). For the MFA and DLA algorithm, the important parameters include k
1 (the number of Neighbor Measurements of the Same Class), k
2 (the number of Neighbor Measurements of Different Classes). For the proposed FA-LPP, the important parameters include k (the number of Neighbor Measurements of the Same Class).Each experiment is repeated 10 times and the average classification rate on ORL and Yale database is listed in Tables2 and 3, which show the superiority of our algorithm. The LPP algorithm only utilizes the nearest neighborhood for partition, while MFA, DLA, and FA-LPP algorithms consider not only the nearest neighborhood but also the class information. Thus, the classification accuracies of MFA, DLA, and FA-LPP algorithms are higher than LPP algorithm. Since the weighting scheme is also taken into account for FA-LPP, the classification accuracy of FA-LPP is higher than MFA and DLA. What is more, there is only one parameter of nearest neighborhood to design for FA-LPP algorithm.Table 2
Best recognition rates of four algorithms on the testing sets of ORL.
G
3
/
P
7
G
4
/
P
6
G
5
/
P
5
LPP
85.0% (60)
89.6% (141)
96.0% (158)
MFA
89.3% (39, 2, 2)
91.4% (46, 2, 1)
96.0% (42, 4, 2)
DLA
94% (42, 2, 2)
92.5% (53, 2, 1)
95.4% (40, 4, 2)
FA-LPP
94.5% (39, 2)
92% (45, 2)
96.6% (40, 4)
For LPP, the numbers in the parentheses are the selected subspace dimensions. For FA-LPP, the first numbers in the parentheses are the selected subspace dimensions and the second numbers are the parameters. For MFA and DLA, the first numbers in the parentheses are the selected subspace dimensions and the second and the third numbers are the parametersk
1 and k
2, respectively.Table 3
Best recognition rates of four algorithms on the testing sets of Yale.
G
3
/
P
8
G
5
/
P
6
G
7
/
P
4
LPP
63.5% (13)
72.3% (14)
80.5% (14)
MFA
61.0% (13, 2, 2)
74.5% (14, 2, 1)
79.7% (21, 4, 2)
DLA
66.8% (15, 2, 2)
78.9% (23, 2, 1)
81.2% (13, 4, 2)
FA-LPP
73.2% (14, 2)
79.5% (16, 2)
85.4% (15, 4)
For LPP, the numbers in the parentheses are the selected subspace dimensions. For FA-LPP, the first numbers in the parentheses are the selected subspace dimensions and the second numbers are the parameters. For MFA and DLA, the first numbers in the parentheses are the selected subspace dimensions and the second and the third numbers are the parametersk
1 and k
2, respectively.


### 4.3. Comprehensive Experiments

In order to comprehensively analyze the performance of FA-LDA and FA-LPP, we conducted facial age group classification experiment and compare it with LDA, WLDA1, WLDA2, LPP, and DLA. In this experiment, the images for age group classification are collected from multiple data sources like 1002 facial images from FG-NET database, 500 images from Google database, and 600 images from the scanned photographs leading to a total of 2102 sample facial images. A few of them are shown in Figure5. For experimental purposes the total 2102 images are split into two training and test sets. Training set consists of 700 images from FG-Net aging database, 300 images from Google, and 400 images collected from scanned images leading to a total of 1400 images. Testing set consists of 302 images from FG-Net database, 200 images from Google, and 200 images from scanned photographs, totaling 702 images.Figure 5
Sample images of FG-Net aging database (the same person with face images of different ages).For the four-class classification, faces in the training dataset will be assigned to age groups according to their labeled age. As every face has an age value labeled, it can provide necessary data for building the factors analysis framework.In this experiment, the Gabor feature was used first in original facial image, and then we used the FA-LDA, FA-LPP, LDA, WLDA1, WLDA2, LPP, and DLA algorithms in feature extraction and dimensionality reduction. The comparative results in Figure6 show that our methods (FA-LDA, FA-LPP) outperformed others. From this result, the classification accuracies of DLA and our FA-LDA and FA-LPP are higher than LDA, WLDA1, WLDA2, and LPP algorithms. The reason is that LDA and LPP are used for linear dimensional reduction without considering weight design problems, respectively, and WLDA1 and WLDA2 are not considered partition design. Furthermore, since LPP is based on manifold learning which is suitable for nonlinear problem such as face age group classification, therefore, the LPP and FA-LPP are better than LDA and FA-LDA, respectively.Figure 6
Comparisons of the performance of different approaches on facial age database.Recently, deep learning is a new area of machine learning research which has got hot attention [25, 26]. Our factors analysis framework can be embedded in deep learning based object recognition system as feature dimension reduction part. Therefore, our factors analysis framework can be seamlessly integrated with the deep learning method.


## 4.1. FA-LDA

To evaluate the proposed FA-LDA algorithm, we compare it with the LDA algorithm on artificial data and real-world face dataset. During the testing phases, the nearest neighbor (NN) rule was used in classification. Note that [24] proposed a probabilistic graphical model framework to mimic the process of generating a face image. However, they focus on addressing the obstacles of small sample set, occlusion, and illumination variations in one sample face identification. So we do not compare our algorithm with [24].Experiments on Artificial Data. The artificial data includes 3 subsets with 3, 4, and 5 classes, respectively, and the prior probability of each class is identical. Each class contains 60 samples with 9 dimensions. There is one class easy to discriminate in each subset. Thus, its within class covariance matrix plays a dominant role compared to other classes. Meanwhile, assume that the other classes obey the normal distribution and have identical within class covariance matrix. In experiments, 10 samples are selected as train set and the other 50 samples are equally divided into 10 groups as test set. The experiments of each class are conducted 10 times and each experiment is repeated 10 times. The classification rate on average and variance of the two algorithms are listed in Table 1. It can be seen that the classification performance of LDA algorithm is not good on this artificial data, which indicates the limitation of LDA. FA-LDA is superior to LDA in classification property. Since LDA algorithm simply uses the average of the within class scatter matrix as unified covariance matrix, it leads to an overemphasis on the scatter matrix of the class easy to discriminate. However, for FA-LDA algorithm the weights of this easily separated class are small and more attention is on the confusable classes.Table 1
Classification accuracies of LDA and FA-LDA on the artificial database.
3 classes
4 classes
5 classes
FA-LDA
LDA
FA-LDA
LDA
FA-LDA
LDA
83.5 ± 7.45
62 ± 8.34
75.5 ± 5.6
38.54 ± 10.21
78.26 ± 5.4
45.26 ± 7.85
81 ± 9.01
56.7 ± 7.45
73 ± 6.68
45.85 ± 7.37
75.66 ± 6.28
46.15 ± 8.21
83.5 ± 9.07
59.5 ± 10.53
73.5 ± 5.62
46.33 ± 5.36
75.5 ± 7.25
39.54 ± 9.25
80.5 ± 8.67
57 ± 9.57
76.67 ± 5.6
42.64 ± 7.6
79.45 ± 5.5
48.26 ± 6.51
81 ± 9.01
53.5 ± 12.45
78.56 ± 5.21
43.85 ± 7.49
76.5 ± 7.50
44.58 ± 9.21
81.5 ± 6.26
61.5 ± 9.57
78.85 ± 5.34
40.28 ± 6.35
70.57 ± 9.14
39.83 ± 5.68
84.5 ± 6.58
60 ± 6.7
80.21 ± 8.63
42.84 ± 5.49
75.64 ± 5.60
46.25 ± 7.35
82.5 ± 8.08
59.5 ± 11.56
78.26 ± 5.23
41.45 ± 8.16
76.42 ± 5.60
42.35 ± 5.36
81 ± 8.45
64.5 ± 6.87
75.95 ± 6.72
43.58 ± 6.53
77.38 ± 6.58
46.25 ± 8.24Experiments on Real-World Data. The ORL database contains 400 images of 40 individuals which is used for real-world experiment. The images were captured at different times and with different variations including expression and facial details, which can be seen from Figure 2. In order to evaluate the superiority of FA-LDA, we compare it with the LDA and current weighted algorithms, in which WLDA1 [17] is based on Euclidean distance and WLDA2 [19] is based on confusion matrix. For training, we randomly selected different numbers (three, four, five, six, seven, eight, and nine) of images per individual and used the rest of the images for testing. Such a trial was independently performed 10 times, and then the average recognition results were calculated. The results are shown in Figure 3. The classification property of LDA is the worst, which illustrated the validity the weighting scheme based on factor analysis criterion. FA-LDA algorithm achieves the best performance for it not only weights S
b to reduce the influence of the far away class, but also weights S
w to improve the anti-interference ability of discriminant criterion.Figure 2
The sample facial images of ORL database.Figure 3
Best recognition rates of four algorithms on the testing sets of ORL.


## 4.2. FA-LPP

To evaluate the proposed FA-LPP algorithm, we compare it with LPP and MFA/DLA on ORL and Yale database. The Yale database is built by computational vision and control center at Yale University. Everyone has 11 images varying in expressions (blinking, glad, sad, or surprised), illumination conditions (frontal, left, or right), and facial details (glasses or no glasses), which can be seen from Figure4.Figure 4
The sample facial images of Yale database.For the ORL and Yale database, the image set is partitioned into the different gallery and probe sets whereG
m
/
P
n indicates that m images per person are randomly selected for training and the remaining n images are used for testing. For the LPP, the important parameters include k (the number of Neighbor Measurements). For the MFA and DLA algorithm, the important parameters include k
1 (the number of Neighbor Measurements of the Same Class), k
2 (the number of Neighbor Measurements of Different Classes). For the proposed FA-LPP, the important parameters include k (the number of Neighbor Measurements of the Same Class).Each experiment is repeated 10 times and the average classification rate on ORL and Yale database is listed in Tables2 and 3, which show the superiority of our algorithm. The LPP algorithm only utilizes the nearest neighborhood for partition, while MFA, DLA, and FA-LPP algorithms consider not only the nearest neighborhood but also the class information. Thus, the classification accuracies of MFA, DLA, and FA-LPP algorithms are higher than LPP algorithm. Since the weighting scheme is also taken into account for FA-LPP, the classification accuracy of FA-LPP is higher than MFA and DLA. What is more, there is only one parameter of nearest neighborhood to design for FA-LPP algorithm.Table 2
Best recognition rates of four algorithms on the testing sets of ORL.
G
3
/
P
7
G
4
/
P
6
G
5
/
P
5
LPP
85.0% (60)
89.6% (141)
96.0% (158)
MFA
89.3% (39, 2, 2)
91.4% (46, 2, 1)
96.0% (42, 4, 2)
DLA
94% (42, 2, 2)
92.5% (53, 2, 1)
95.4% (40, 4, 2)
FA-LPP
94.5% (39, 2)
92% (45, 2)
96.6% (40, 4)
For LPP, the numbers in the parentheses are the selected subspace dimensions. For FA-LPP, the first numbers in the parentheses are the selected subspace dimensions and the second numbers are the parameters. For MFA and DLA, the first numbers in the parentheses are the selected subspace dimensions and the second and the third numbers are the parametersk
1 and k
2, respectively.Table 3
Best recognition rates of four algorithms on the testing sets of Yale.
G
3
/
P
8
G
5
/
P
6
G
7
/
P
4
LPP
63.5% (13)
72.3% (14)
80.5% (14)
MFA
61.0% (13, 2, 2)
74.5% (14, 2, 1)
79.7% (21, 4, 2)
DLA
66.8% (15, 2, 2)
78.9% (23, 2, 1)
81.2% (13, 4, 2)
FA-LPP
73.2% (14, 2)
79.5% (16, 2)
85.4% (15, 4)
For LPP, the numbers in the parentheses are the selected subspace dimensions. For FA-LPP, the first numbers in the parentheses are the selected subspace dimensions and the second numbers are the parameters. For MFA and DLA, the first numbers in the parentheses are the selected subspace dimensions and the second and the third numbers are the parametersk
1 and k
2, respectively.


## 4.3. Comprehensive Experiments

In order to comprehensively analyze the performance of FA-LDA and FA-LPP, we conducted facial age group classification experiment and compare it with LDA, WLDA1, WLDA2, LPP, and DLA. In this experiment, the images for age group classification are collected from multiple data sources like 1002 facial images from FG-NET database, 500 images from Google database, and 600 images from the scanned photographs leading to a total of 2102 sample facial images. A few of them are shown in Figure5. For experimental purposes the total 2102 images are split into two training and test sets. Training set consists of 700 images from FG-Net aging database, 300 images from Google, and 400 images collected from scanned images leading to a total of 1400 images. Testing set consists of 302 images from FG-Net database, 200 images from Google, and 200 images from scanned photographs, totaling 702 images.Figure 5
Sample images of FG-Net aging database (the same person with face images of different ages).For the four-class classification, faces in the training dataset will be assigned to age groups according to their labeled age. As every face has an age value labeled, it can provide necessary data for building the factors analysis framework.In this experiment, the Gabor feature was used first in original facial image, and then we used the FA-LDA, FA-LPP, LDA, WLDA1, WLDA2, LPP, and DLA algorithms in feature extraction and dimensionality reduction. The comparative results in Figure6 show that our methods (FA-LDA, FA-LPP) outperformed others. From this result, the classification accuracies of DLA and our FA-LDA and FA-LPP are higher than LDA, WLDA1, WLDA2, and LPP algorithms. The reason is that LDA and LPP are used for linear dimensional reduction without considering weight design problems, respectively, and WLDA1 and WLDA2 are not considered partition design. Furthermore, since LPP is based on manifold learning which is suitable for nonlinear problem such as face age group classification, therefore, the LPP and FA-LPP are better than LDA and FA-LDA, respectively.Figure 6
Comparisons of the performance of different approaches on facial age database.Recently, deep learning is a new area of machine learning research which has got hot attention [25, 26]. Our factors analysis framework can be embedded in deep learning based object recognition system as feature dimension reduction part. Therefore, our factors analysis framework can be seamlessly integrated with the deep learning method.


## 5. Conclusions

In this paper, we aim to provide insights into the relationship among the state-of-the-art dimensionality reduction algorithms as well as to facilitate the design of new algorithms. A general framework known as factor analysis, along with its linearization, kernelization, and tensorization, has been proposed to provide a unified perspective for the understanding and comparison of many popular dimensionality reduction algorithms. Moreover, the factor analysis framework can be used as a general platform to develop new algorithms for dimensionality reduction. As shown in this paper, we have proposed two novel dimensionality reduction algorithms called FA-LDA and FA-LPP by designing the objective function of factors separation that characterize the weight matrix and the partition and by optimizing their corresponding criteria based on the factor analysis framework. These new algorithms are shown to effectively overcome the data distribution assumption of the traditional LDA and LPP algorithm. Thus, FA-LDA and FA-LPP are more general algorithms for discriminant analysis.A byproduct of this paper is a series of kernelization and tensorization versions of the factors analysis. One of our future works is to systematically compare all possible extensions of the algorithms mentioned in this paper.



---

*Source: 9347838-2016-04-26.xml*