# Sparse Graph Embedding Based on the Fuzzy Set for Image Classification

Authors: Minghua Wan; Mengting Ge; Tianming Zhan; Zhangjing Yang; Hao Zheng; Guowei Yang
Journal: Complexity
 (2021)

---

## Abstract

In recent years, many face image feature extraction and dimensional reduction algorithms have been proposed for linear and nonlinear data, such as local-based graph embedding algorithms or fuzzy set algorithms. However, the aforementioned algorithms are not very effective for face images because they are always affected by overlaps (outliers) and sparsity points in the database. To solve the problems, a new and effective dimensional reduction method for face recognition is proposed—sparse graph embedding with the fuzzy set for image classification. The aim of this algorithm is to construct two new fuzzy Laplacian scattering matrices by using the local graph embedding and fuzzyk-nearest neighbor. Finally, the optimal discriminative sparse projection matrix is obtained by adding elastic network regression. Experimental results and analysis indicate that the proposed algorithm is more effective than other algorithms in the UCI wine dataset and ORL, Yale, and AR standard face databases.


---

## Body

## 1. Introduction

Machine learning has become one of the main research tools for small sample learning in complex networks. Using the ability of small sample and the knowledge learned, it is helpful to improve the model of key participant discovery in reality and effectively find the key participants under different objective criteria. There are more and more manifold learning algorithms with local preserving ability, which have become the active research fields in machine learning and pattern recognition [1–6]. Among them, the classical algorithms such as Laplacian eigenmaps (LE) [7, 8], isometric feature mapping (ISOMAP) [9], local linear embedding (LLE) [10, 11], and local tangent space alignment (LTSA) [12] are widely applied in machine learning and pattern recognition. LLE algorithm creates local optimal coordinates with least construction cost to map them to global coordinates. However, LLE algorithm lacks the generalization ability when applied to new face samples. Therefore, neighborhood preserving embedding (NPE) [13] was proposed by He et al., which maintained the optimal local neighborhood reconstruction relationship on the data. Another representative learning method of local preserving projection (LPP) [14, 15] is an unsupervised algorithm derived from LE, where local information can be found and retained in an embedding space. Recently, in order to unify the manifold learning framework, linear graph embedding (LGE) algorithm was proposed by Yan et al. [16].However, in the real world, the face image is always affected by various expressions, posture, and illumination variation, so LGE and its extended algorithms cannot effectively solve the above problems [17–19]. By using the fuzzy k-nearest neighbor of fuzzy set [20] technology, a lot of research studies have been completed on fuzzy image filtration, fuzzy edge detection, and fuzzy image segmentation. Its ultimate goal is to solve the uncertainty in pattern recognition and machine learning [21–24].However, all the above methods are difficult to get a reasonable explanation, that is, feature extraction and classification; feature plays an important role in high-dimensional data [25–27]. Many sparse subspace learning methods have aroused great research interest [28–30] in recent years. In recent years, the more effective sparse feature extraction methods include the least absolute shrinkage and selection operator (LASSO) [31], LARS [32], and elastic net [33]. Under the L1-norm, these methods can make the corresponding coefficient of the partial characteristics shrink to zero [24–38].In order to solve the above problems, we study the LGE algorithm based on subspace learning, elastic network regression, and fuzzy set theory, namely, sparse graph embedding with fuzzy set for image classification. In the proposed method, fuzzyk-nearest neighbor (FKNN) and sparse learning are used to improve the discrimination ability of the method in a low-dimensional space. Firstly, the value of the membership matrix is calculated by FKNN, and then the value is introduced into the Laplacian scatter matrix of within-class and between-class, respectively.The novelty of our algorithm includes the following perspectives:(1)
In this algorithm, manifold learning method and fuzzy set are used to keep the original neighbor relations and the intrinsic geometry structure for overlapped (outliers) data points. At the same time, the algorithm uses FKNN to construct two new fuzzy graphs, which has stronger recognition ability compared with other graph-based algorithms.(2)
The proposed algorithm has a lower sensitivity to sparsity points in the data through elastic network regression which enhances the robustness of our algorithm. The sparse projections learned by the proposed algorithm indicate physical interpretation in real-world applications.The rest of this paper is planned as follows: the idea of linear methods is described in Section2. The algorithm innovation and implementation are reviewed in Section 3. The experimental results of the proposed algorithm verify its discrimination and effectiveness on four different standard databases in Section 4. The conclusions are summarized in Section 5.


## 2. Outlines of FKNN, LLE, and Lasso Regression Net

Firstly, we considerN set samples X=x1,x2,…,xN of linear transformation mapping original space xi∈RD into feature space Y=y1,y2,…,yN∈Rd, where n>d. The new eigenvector yi∈Rd is from the linear transformation formula:(1)yi=PTxi,i=1,…,N,where P∈Rn×d is the linear transformation projection matrix. The algorithms of FKNN, LLE, and lasso regression network are reviewed, respectively.


### 2.1. Fuzzyk-Nearest Neighbor (FKNN)

In the FKNN algorithm, the value of the membership degree is generated by the class assignment of the fuzzyk-nearest neighbor and the corresponding degrees of class membership. The following five steps comprise the FKNN algorithm:Step 1:
the value of the Euclidean distance matrix, which is calculated between eigenvectors using the training sample set.Step 2:
the value of the diagonal element is set to infinity by using the Euclidean distance matrix.Step 3:
the value of the distance matrix in the ascending order (process each column separately).Step 4:
the class corresponding to the set pattern, which is in the nearest neighborhood of the pattern sample (becausek neighborhoods are concerned, this returns a list of k integers).Step 5:
the expression proposed in [20], which is used to calculate the type i membership of mode j, which is(2)uij=0.51+0.49×nijK,ifjthsample∈ithclass,0.49×nijK,otherwise,wherenij is the jth sample (patterns) inclusion of the ith class. For example, if nij has neighbors belonging to the equal pattern, the member value will remain close to 0.51. Otherwise, it will remain close to 0.49. Generally, uij contains the following important properties:(3)∑i=1cuij=1,0<∑j=1Nuij<N.Therefore,U can be obtained by the FKNN algorithm:(4)U=uij,i=1,2,…,c,j=1,2,…N,wherec is the total class number and N is all sample numbers in the ith class.


### 2.2. Locally Linear Embedding

LLE is a classic method of manifold learning methods, which can better maintain the original manifold structure after dimensionality reduction. There are three main steps to the algorithm.The first step is to calculate thek-nearest neighbor relation of each data point xi by Euclidean distance.The second step is to count the coefficient ofW=wijN×N by minimizing the reconstruction error. We can get the reconstruction weight matrix W:(5)minJLW=∑i=1Nxi−∑j=1Nwijxj2.The reconstruction errorxi is transformed into the following equation:(6)ξi=xi−∑j=1Nwijxj2=∑j=1Nwijxi−xj2=∑j=1Nwijxi−xj∑t=1Nwitxi−xt=∑j=1N∑t=1NwijwitGjti,where Gjtj=xi−xjTxi−xt is the local Gram matrix. Optimal coefficient with ∑j=1Nwij=1 constraint is obtained using the least square problem to solve(7)wij=∑t=1NGijt−1∑p=1N∑q=1NGipq−1.After iterating steps 1 and 2 forN times, we get the reconstruction weight coefficient matrix W=wijN×N.The third step is to use weight matrixW to reconstruct yi in equation (7).By minimizing the reconstruction error function, the inherent geometric features of the embedded data can be maintained. Considering the transformationyi=PTxi, the objective function is simplified as follows:(8)minJLP=∑i=1Nyi−∑j=1Nwijyj2=trPTXMXTP,where M=I−WTI−W.


### 2.3. Lasso Regression Network

In this study, we use the lasso algorithm to obtain the optimal solution for sparse samples. In this algorithm, a penalty function is constructed to obtain the biased estimation of complex linear data processing. Using the lasso interpretation model, the absolute value of the regression coefficient minimizes the sum of the squares of the residuals so that the regression coefficient is exactly equal to 0.Suppose we have the value ofXi=xi1,…,xipT and the value of Xi,yi,i=1,2,…,N in the ith observation variable.(9)yi=αi+∑j=1pβjxij+εi,where xi and yi are variables and p is the number of elements in xi.In order to further elaborate and solve this problem, it is assumed that the response variablesyi are independent of the given observation conditions.In general, the regression structure is as follows:(10)1N∑jxij=0,1N∑jxij2=0.The available lasso estimate is(11)α^,β^=argmin∑iyi−αi−∑jβjxij2,s.t.∑jβj≤j.Here,t≥0 is a harmonic parameter, and α:α^=y¯ is obtained for all t. Let t0=∑jβj,t≤t0, and the number of regression coefficients will decrease to 0. If the harmonic parameter t is controlled, the regression coefficient can be reduced.


## 2.1. Fuzzyk-Nearest Neighbor (FKNN)

In the FKNN algorithm, the value of the membership degree is generated by the class assignment of the fuzzyk-nearest neighbor and the corresponding degrees of class membership. The following five steps comprise the FKNN algorithm:Step 1:
the value of the Euclidean distance matrix, which is calculated between eigenvectors using the training sample set.Step 2:
the value of the diagonal element is set to infinity by using the Euclidean distance matrix.Step 3:
the value of the distance matrix in the ascending order (process each column separately).Step 4:
the class corresponding to the set pattern, which is in the nearest neighborhood of the pattern sample (becausek neighborhoods are concerned, this returns a list of k integers).Step 5:
the expression proposed in [20], which is used to calculate the type i membership of mode j, which is(2)uij=0.51+0.49×nijK,ifjthsample∈ithclass,0.49×nijK,otherwise,wherenij is the jth sample (patterns) inclusion of the ith class. For example, if nij has neighbors belonging to the equal pattern, the member value will remain close to 0.51. Otherwise, it will remain close to 0.49. Generally, uij contains the following important properties:(3)∑i=1cuij=1,0<∑j=1Nuij<N.Therefore,U can be obtained by the FKNN algorithm:(4)U=uij,i=1,2,…,c,j=1,2,…N,wherec is the total class number and N is all sample numbers in the ith class.


## 2.2. Locally Linear Embedding

LLE is a classic method of manifold learning methods, which can better maintain the original manifold structure after dimensionality reduction. There are three main steps to the algorithm.The first step is to calculate thek-nearest neighbor relation of each data point xi by Euclidean distance.The second step is to count the coefficient ofW=wijN×N by minimizing the reconstruction error. We can get the reconstruction weight matrix W:(5)minJLW=∑i=1Nxi−∑j=1Nwijxj2.The reconstruction errorxi is transformed into the following equation:(6)ξi=xi−∑j=1Nwijxj2=∑j=1Nwijxi−xj2=∑j=1Nwijxi−xj∑t=1Nwitxi−xt=∑j=1N∑t=1NwijwitGjti,where Gjtj=xi−xjTxi−xt is the local Gram matrix. Optimal coefficient with ∑j=1Nwij=1 constraint is obtained using the least square problem to solve(7)wij=∑t=1NGijt−1∑p=1N∑q=1NGipq−1.After iterating steps 1 and 2 forN times, we get the reconstruction weight coefficient matrix W=wijN×N.The third step is to use weight matrixW to reconstruct yi in equation (7).By minimizing the reconstruction error function, the inherent geometric features of the embedded data can be maintained. Considering the transformationyi=PTxi, the objective function is simplified as follows:(8)minJLP=∑i=1Nyi−∑j=1Nwijyj2=trPTXMXTP,where M=I−WTI−W.


## 2.3. Lasso Regression Network

In this study, we use the lasso algorithm to obtain the optimal solution for sparse samples. In this algorithm, a penalty function is constructed to obtain the biased estimation of complex linear data processing. Using the lasso interpretation model, the absolute value of the regression coefficient minimizes the sum of the squares of the residuals so that the regression coefficient is exactly equal to 0.Suppose we have the value ofXi=xi1,…,xipT and the value of Xi,yi,i=1,2,…,N in the ith observation variable.(9)yi=αi+∑j=1pβjxij+εi,where xi and yi are variables and p is the number of elements in xi.In order to further elaborate and solve this problem, it is assumed that the response variablesyi are independent of the given observation conditions.In general, the regression structure is as follows:(10)1N∑jxij=0,1N∑jxij2=0.The available lasso estimate is(11)α^,β^=argmin∑iyi−αi−∑jβjxij2,s.t.∑jβj≤j.Here,t≥0 is a harmonic parameter, and α:α^=y¯ is obtained for all t. Let t0=∑jβj,t≤t0, and the number of regression coefficients will decrease to 0. If the harmonic parameter t is controlled, the regression coefficient can be reduced.


## 3. Sparse Graph Embedding with Fuzzy Set

### 3.1. The Basic Idea of the Proposed Algorithm

Firstly, two new fuzzy graphs are constructed to keep the same data points together and different data points separate by the FKNN algorithm. Secondly, in order to avoid singularity, the algorithm seeks the difference between the fuzzy scatter matrix of within-class and between-class by the MMC [39] framework. Finally, the optimal mapping projection matrix is acquired by the lasso regression network.


### 3.2. Fuzzy Within-Class Scatter Matrix with Locality Preserving

Firstly, we minimize the fuzzy within-class scatter matrix while maintaining the locality preserving of each data pointxi and reconstruct the data points by using the linear coefficients of other data points. The algorithm is divided into four steps:(1)
TheKc-nearest neighbors of xi are obtained by Euclidean distance.(2)
The reconstruction weight matrixW=wkjN×N is minimized by(12)minJLW=∑i=1Nxi−∑j=1Kcwijxj2,wherewkj=0. If xk is not the neighbor of xj, the row sum of W is 1: ∑j=1Kcwkj=1.We getW=wkjN×N after repeating steps 1 and 2 forN times.(3)
To calculate the fuzzy weight of affinity, we can specify the new weight matrixWijG:(13)WijG=uijG⋅∗W,whereuijG=0.51+0.49×nij/Kc0.49×nij/Kc and “∗” denotes the matrix element-wise multiplication.(4)
The reconstruction error function is used to reconstruct weight matrixWG expressed as yi:(14)minJLY=∑i=1Nyi−∑j=1NWijGyj2,whereyj is the neighbor data point of yi and yi is the output of xi.The objective function can be obtained by using equation (1) as follows:(15)JLV=∑i=1Nyi−∑j=1NWijGyj2=tr∑i=1Nyi−∑j=1NWijGyjyi−∑j=1NWijGyjT=tr∑i=1Nyi−∑j=1NWijGyjyi−∑j=1NWijGyjT=trYI−WGTI−WGTTYT=trYI−WGTI−WGYT=trPTXMXTP,where matrix M=I−WGTI−WG.


### 3.3. Fuzzy Between-Class Scatter Matrix with Locality Preserving

First, if they have different labels, we output the sum of the maximum square distances between the data. Therefore, the following three steps are used to solve the fuzzy between-class scatter matrix with locality preserving:(1)
The similarity matrixWijp is constructed as follows:(16)Wijp=1,xi∈NKp+xj,0,otherwise,whereNKp+xj indicates that Kp is the nearest neighbor of sample data point Xj in the different class.(2)
To calculate the fuzzy weight of affinity, we can specify the new weight matrixKp:(17)WijG′=uijG′⋅∗Wijp,whereuijG′=0.51+0.49×nij/KP0.49×nij/KP.(3)
Complete the final embedding:(18)maxJGY=∑i=1N∑j=1Nyi−yj2WijG′.By considering the mapping relation in equation (1), the objective function can be simplified:(19)JGV=∑i∑jyi−yj2WijG′=∑i∑jPTxi−PTxj2WijG′=2PTXDp−WpXTP=2PTXLpXTP.


### 3.4. The Criteria of the Proposed Algorithm

Finally, the MMC algorithm framework is used to minimize fuzzy within-class scatter matrixJLV and maximize fuzzy between-class scatter matrix JGV to find the optimal projection matrix P. In fact, we can get such optimal projection matrix P through the following multiobjective optimization problems, namely,(20)mintrPTXMXTPmaxtrPTXLpXTPs.t.PTXXTP=I.Finally, the above formula can be transformed into a constraint problem:(21)mintrPTXM−μLpXTPs.t.PTXXTP=I,Where μ>0 is a constant used to balance matrix M and matrix Lp.The Lagrange multiplier method can be used in equation (21):(22)LP,λ=PTXM−μLpXTP−ηPTXXTP−I=0,where η is the Lagrangian multiplier parameter. Thus, the above formula can be further simplified to obtain(23)XM−μLpXTP=λiXXTP,where Pi is a generalised eigenvector corresponding to a generalised eigenvalue λi.Projection matrixP can be obtained as follows:(24)XM−μLpXTP=λiXXTP,CardP=K,where CardP=K≤n indicates the quantity of nonzero elements in matrix P. In this way, the optimal sparse projection matrix P=p1,p2,…,pd can be obtained in equation (24).At last, we need the following theorem of feature analysis. After that, the learning problem of the sparse projection function is transformed into the regression framework.Theorem 1.
Lety be the eigenvector of the following formula with eigenvalue λ:(25)M−μLpy=λy.IfXTp=y, then v is the eigenvector of the eigenproblem with the same eigenvalue λ in equation (23).Proof.
SinceXTp=y, it is easy to prove the following formula according to equation (23):(26)XM−μLpy=λXy⇒XM−μLpXTp=λXXTp.
Therefore,p is the eigenvector of the same eigenvalue of equation (23).
We can solve this problem with ridge regression:(27)P=argminp∑i=1mpTxi−y¯i2+α∑j=1nv¯j2,where α≥0. p¯j is the jth element of v, and y¯i is the ith element of y.
However, the ridge penalty of equation (27) does not supply sparse matrix pi. When using the lasso regression algorithm, add L1-norm to vi to get(28)P=argmin∑i=1mpiTxi−yi2+β∑j=1npj.
So, we integrate the lasso regression and ridge regression [40]:(29)P=argmin∑i=1mpiTxi−yi2+α∑j=1npj¯2+β∑j=1npj.
Then, we get the optimal sparse transformation matrixP=p1,p2,…,pl.


## 3.1. The Basic Idea of the Proposed Algorithm

Firstly, two new fuzzy graphs are constructed to keep the same data points together and different data points separate by the FKNN algorithm. Secondly, in order to avoid singularity, the algorithm seeks the difference between the fuzzy scatter matrix of within-class and between-class by the MMC [39] framework. Finally, the optimal mapping projection matrix is acquired by the lasso regression network.


## 3.2. Fuzzy Within-Class Scatter Matrix with Locality Preserving

Firstly, we minimize the fuzzy within-class scatter matrix while maintaining the locality preserving of each data pointxi and reconstruct the data points by using the linear coefficients of other data points. The algorithm is divided into four steps:(1)
TheKc-nearest neighbors of xi are obtained by Euclidean distance.(2)
The reconstruction weight matrixW=wkjN×N is minimized by(12)minJLW=∑i=1Nxi−∑j=1Kcwijxj2,wherewkj=0. If xk is not the neighbor of xj, the row sum of W is 1: ∑j=1Kcwkj=1.We getW=wkjN×N after repeating steps 1 and 2 forN times.(3)
To calculate the fuzzy weight of affinity, we can specify the new weight matrixWijG:(13)WijG=uijG⋅∗W,whereuijG=0.51+0.49×nij/Kc0.49×nij/Kc and “∗” denotes the matrix element-wise multiplication.(4)
The reconstruction error function is used to reconstruct weight matrixWG expressed as yi:(14)minJLY=∑i=1Nyi−∑j=1NWijGyj2,whereyj is the neighbor data point of yi and yi is the output of xi.The objective function can be obtained by using equation (1) as follows:(15)JLV=∑i=1Nyi−∑j=1NWijGyj2=tr∑i=1Nyi−∑j=1NWijGyjyi−∑j=1NWijGyjT=tr∑i=1Nyi−∑j=1NWijGyjyi−∑j=1NWijGyjT=trYI−WGTI−WGTTYT=trYI−WGTI−WGYT=trPTXMXTP,where matrix M=I−WGTI−WG.


## 3.3. Fuzzy Between-Class Scatter Matrix with Locality Preserving

First, if they have different labels, we output the sum of the maximum square distances between the data. Therefore, the following three steps are used to solve the fuzzy between-class scatter matrix with locality preserving:(1)
The similarity matrixWijp is constructed as follows:(16)Wijp=1,xi∈NKp+xj,0,otherwise,whereNKp+xj indicates that Kp is the nearest neighbor of sample data point Xj in the different class.(2)
To calculate the fuzzy weight of affinity, we can specify the new weight matrixKp:(17)WijG′=uijG′⋅∗Wijp,whereuijG′=0.51+0.49×nij/KP0.49×nij/KP.(3)
Complete the final embedding:(18)maxJGY=∑i=1N∑j=1Nyi−yj2WijG′.By considering the mapping relation in equation (1), the objective function can be simplified:(19)JGV=∑i∑jyi−yj2WijG′=∑i∑jPTxi−PTxj2WijG′=2PTXDp−WpXTP=2PTXLpXTP.


## 3.4. The Criteria of the Proposed Algorithm

Finally, the MMC algorithm framework is used to minimize fuzzy within-class scatter matrixJLV and maximize fuzzy between-class scatter matrix JGV to find the optimal projection matrix P. In fact, we can get such optimal projection matrix P through the following multiobjective optimization problems, namely,(20)mintrPTXMXTPmaxtrPTXLpXTPs.t.PTXXTP=I.Finally, the above formula can be transformed into a constraint problem:(21)mintrPTXM−μLpXTPs.t.PTXXTP=I,Where μ>0 is a constant used to balance matrix M and matrix Lp.The Lagrange multiplier method can be used in equation (21):(22)LP,λ=PTXM−μLpXTP−ηPTXXTP−I=0,where η is the Lagrangian multiplier parameter. Thus, the above formula can be further simplified to obtain(23)XM−μLpXTP=λiXXTP,where Pi is a generalised eigenvector corresponding to a generalised eigenvalue λi.Projection matrixP can be obtained as follows:(24)XM−μLpXTP=λiXXTP,CardP=K,where CardP=K≤n indicates the quantity of nonzero elements in matrix P. In this way, the optimal sparse projection matrix P=p1,p2,…,pd can be obtained in equation (24).At last, we need the following theorem of feature analysis. After that, the learning problem of the sparse projection function is transformed into the regression framework.Theorem 1.
Lety be the eigenvector of the following formula with eigenvalue λ:(25)M−μLpy=λy.IfXTp=y, then v is the eigenvector of the eigenproblem with the same eigenvalue λ in equation (23).Proof.
SinceXTp=y, it is easy to prove the following formula according to equation (23):(26)XM−μLpy=λXy⇒XM−μLpXTp=λXXTp.
Therefore,p is the eigenvector of the same eigenvalue of equation (23).
We can solve this problem with ridge regression:(27)P=argminp∑i=1mpTxi−y¯i2+α∑j=1nv¯j2,where α≥0. p¯j is the jth element of v, and y¯i is the ith element of y.
However, the ridge penalty of equation (27) does not supply sparse matrix pi. When using the lasso regression algorithm, add L1-norm to vi to get(28)P=argmin∑i=1mpiTxi−yi2+β∑j=1npj.
So, we integrate the lasso regression and ridge regression [40]:(29)P=argmin∑i=1mpiTxi−yi2+α∑j=1npj¯2+β∑j=1npj.
Then, we get the optimal sparse transformation matrixP=p1,p2,…,pl.


## 4. Experimental Results and Analysis

In order to test and verify the effectiveness of the proposed algorithm, we compared it with PCA [41], LDA [42], MMC [39], LPP [14, 15], LLE [10, 11], SDE [38], and FLGE/MMC [4] on the UCI wine dataset and ORL, Yale, and AR face standard databases, respectively. In our experiments, the values of parameters in our algorithm are given in Table 1. Each experiment runs independently for ten times when different training samples are taken for the training set. In the experiment, we used Euclidean distance and nearest neighbor classifier to measure and classify, respectively. The four databases’ experimental results are completed in the same experimental environment (CPU: P4 4 GHz; RAM: 8 GB).Table 1
Parameter values of our proposed method on different databases.
DatasetThe parameter of our proposed methodTKCKpKRun timesUCI wine48T−12 : 2 : 2010 : 10 : 10010ORL2 : 1 : 6T−12 : 2 : 205 : 5 : 4050Yale2 : 1 : 6T−12 : 2 : 205 : 5 : 4050AR2 : 1 : 6T−15 : 5 : 5010 : 10 : 4010T is training images from each class.


### 4.1. The Experiments and Interpretations in the UCI Wine Database

Now, we use a wine database from the UCI machine learning library to further validate and interpret the proposed method. There are 3 classes, 13 features, and 178 instances in the wine database. We select 48 samples in each class, and the basic statistics of the variance, mean, and range of the 13 features in the experiment are shown in Table2.Table 2
The first sparse projection axis and the corresponding sparse load (k = 5) of the proposed method and USSL.
FeaturesMeanVarianceRangeUSSLThe proposed method113.00.73.80022.31.25.10.130.2832.40.081.900419.511.219.40.090.36599.7204.092.00.02062.30.42.90072.01.04.7−0.99−0.9580.40.020.50091.60.33.200105.15.411.700.19111.00.051.200122.60.52.70013746.999166.714020.01040.0008Both the proposed method and USSL sparse projections are suitable for classification. We learn that there are precise physical explanations, which provide us a deeper understanding of the database. That is to say, the second (malic acid), the fourth (ash alkaline), the fifth (magnesium), the seventh (xanthic acid), the tenth (color intensity), and the thirteenth (proline) are the most important and significant distinguishing features. Moreover, the proposed method consistently outperforms USSL within lower dimension.


### 4.2. The Experiments in the ORL Database

There are 40 classes in the ORL face database [43]; each class contains 10 different face images for a total of 400 face images, among which facial expression, posture, and illumination vary. The images allow the face to tilt and rotate 20 degrees. In addition, there are some changes as high as about 10% in this scale. Figure 1 shows ten images of one class in the ORL face database. All images were normalized to 56 × 46 resolution.Figure 1
Ten images of one class in the ORL face database.In this experiment, the performance of PCA, MMC, LDA, LLE, LPP, SDE, FLGE/MMC, and the new proposed method is tested, respectively. First of all,T images (T ranging from 2 to 6) from each class of the image database are randomly selected as the training sample set. Secondly, the remaining 10−T face images of each class are selected as the testing sample set. Finally, we independently run 50 times on T images of each class.In the PCA phase of seven algorithms (MMC, LDA, LPP, LLE, SDE, FLGE/MMC, and the proposed algorithm), 95% of the face image energy is retained for computational effectiveness. Table3 displays the standard deviation, the best average recognition rates (%), and corresponding feature dimension of various algorithms in the ORL face database.Table 3
The standard deviation, the best average recognition rates (%), and corresponding feature dimension of various algorithms in the ORL face database.
MethodsT23456PCA72.25±2.61 (44)81.62±0.65 (44)83.98±1.00 (46)85.41±1.30 (44)87.41±1.98 (46)LDA76.35±1.07 (38)83.40±1.69 (38)84.13±2.03 (38)86.07±1.16 (38)88.60±0.78 (38)MMC75.01±1.77 (40)83.96±0.48 (38)85.37±2.48 (40)87.47±0.85 (38)90.49±1.17 (36)LPP73.52±0.05 (42)80.28±0.78 (44)89.18±1.42 (44)91.13±0.82 (40)92.12±1.21 (38)LLE69.78±0.82 (46)72.66±0.73 (50)76.32±1.14 (50)78.85±1.15 (50)88.44±1.55 (48)SDE77.29±1.20 (28)81.20±0.27 (28)87.89±1.39 (28)89.24±2.41 (26)90.12±0.68 (30)FLGE/MMC79.30±0.83 (46)88.61±1.79 (42)93.53±1.82 (50)95.84±1.23 (48)97.36±1.24 (50)The proposed method91.75±0.45 (26)93.14±1.08 (26)95.14±1.22 (30)96.30±1.24 (26)98.13±1.23 (24)Figure2 shows the curve when four face images in each class are randomly selected as the training set in the ORL face database. The curve shows the average recognition rate (%) of each algorithm corresponding to different dimensions.Figure 2
The average recognition rates (%) of various algorithms corresponding to various dimensions when four face images of each class are randomly selected as the training set in the ORL face database.


### 4.3. The Experiments in the Yale Database

There are 15 classes in the Yale face database [44]; each class contains 11 different face images, a total of 165 face images, among which normal, surprised, happy, sad, sleepy, and winking are varied. Figure 3 shows 11 images of one class in the Yale face database.Figure 3
Eleven images of one class in the Yale face database.In our experiment, all images were normalized to 50 × 40 resolution in the Yale database. It also retains 95% of face image energy as the same as the ORL face database. First of all,T images (T ranging from 2 to 6) from each class of image database are randomly selected as the training sample set. Secondly, the remaining 11−T face images of each class are selected as the testing sample set. Finally, we independently run 50 times on T images of each class. The curve in Figure 4 shows the average recognition rate (%) of various algorithms under different feature dimensions when six face images of each class are randomly selected as training sets in the Yale face database. Table 4 shows the standard deviation, the best average recognition rate (%), and the corresponding feature dimensions of various algorithms in the Yale face database, respectively.Figure 4
The average recognition rates (%) of various algorithms corresponding to various dimensions when six face images of each class are randomly selected as the training set in the Yale face database.Table 4
The standard deviation, best average recognition rates (%), and corresponding feature dimension of various algorithms in the Yale face database.
MethodsT23456PCA77.29±1.20 (38)80.20±1.27 (40)83.99±1.38 (38)84.72±1.24 (40)86.21±0.80 (46)LDA80.45±1.48 (16)84.55±1.06 (16)87.85±0.45 (16)87.20±1.64 (14)88.54±0.82 (14)MMC80.58±0.71 (12)82.84±0.88 (12)85.87±1.12 (12)86.83±0.37 (12)87.79±0.50 (14)LPP80.40±1.80 (24)85.79±0.38 (22)88.00±0.50 (20)89.57±0.70 (18)91.05±1.23 (26)LLE83.55±1.38 (14)83.58±0.59 (6)85.90±0.75 (6)88.37±1.63 (8)88.97±1.56 (8)SDE85.18±0.70 (48)86.48±0.82 (46)87.85±1.20 (46)90.16±1.45 (48)92.47±1.32 (46)FLGE/MMC90.45±0.75 (46)91.48±1.01 (46)93.21±1.24 (44)95.34±1.21 (42)96.33±0.21 (46)The proposed method91.50±0.55 (36)92.58±0.17 (36)95.53±0.86 (38)97.12±0.37 (38)98.75±0.35 (36)


### 4.4. The Experiments in the AR Database

The AR face database [45] has 70 men and 56 women, a total of 126 people. The database consists of more than 4000 color people, including occluded front views, lighting conditions, and facial expressions. In AR database, there are 120 face images, including 65 male and 55 female. These images are divided into two parts (two weeks apart), each part containing 20 color face images. The face part image of each person is normalized to 50 × 40 pixels. And Figure 5 displays an example image of one class in the AR face database.Figure 5
Twenty images of one person in the AR database. The first and second lines of face images were taken at two-week intervals.It also retains 95% of face image energy as the same as ORL and Yale face database. First of all,T images (T ranging from 2 to 6) from each class of the image database are randomly selected as the training sample set. Secondly, the remaining 20−T face images of each class are selected as the testing sample set. Finally, we independently run 10 times on T images of each class. Table 5 shows the standard deviation, the best average recognition rate (%), and the corresponding feature dimensions of various algorithms in the AR face database, respectively.Table 5
The standard deviation, best average recognition rates (%), and corresponding feature dimension of various algorithms in the AR face database.
MethodsT23456PCA66.68 ± 1.11 (145)70.21 ± 1.62 (145)77.60 ± 1.27 (150)79.03 ± 0.81 (150)81.54 ± 1.04 (145)LDA71.5 ± 0.71 (110)75.58 ± 0.76 (115)82.53 ± 0.81 (120)87.12 ± 0.33 (115)87.58 ± 0.75 (120)MMC69.39 ± 1.15 (120)75.71 ± 0.59 (115)82.98 ± 0.62 (115)85.27 ± 0.94 (120)87.86 ± 0.82 (120)LPP72.82 ± 0.81 (85)78.78 ± 1.10 (85)83.70 ± 0.65 (80)88.75 ± 0.31 (85)89.93 ± 0.48 (70)LLE70.40 ± 0.89 (125)74.31 ± 1.16 (145)83.35 ± 1.49 (140)84.78 ± 0.76 (150)86.98 ± 0.40 (145)SDE74.90 ± 0.08 (85)80.10 ± 1.36 (75)84.35 ± 1.36 (80)89.80 ± 0.65 (85)92.50 ± 1.18 (70)FLGE/MMC75.09 ± 1.49 (75)82.42 ± 1.08 (70)88.23 ± 0.8 (75)90.17 ± 1.36 (75)93.38 ± 1.20 (70)The proposed method79.31 ± 1.65 (80)85.79 ± 0.75 (90)90.86 ± 0.82 (95)92.74 ± 0.95 (85)95.82 ± 0.68 (90)Figure2 shows the curve when five face images in each class are randomly selected as the training set in the AR face database. The curve shows the average recognition rate (%) of each algorithm corresponding to different dimensions.


### 4.5. Overall Observations and Discussion

After analyzing the experimental results of the above four databases, the following conclusions can be drawn:(1)
As can be seen from Table1, the first six important characteristics (the second, fourth, fifth, seventh, tenth, and thirteenth characteristics) reflect the data distribution. The last characteristic factor plays a decisive role in the data distribution because of its wide range and large variance.(2)
It can be seen from Tables3–5 that compared with PCA, MMC, LDA, LLE, LPP, SDE, and FLGE/MMC, the proposed algorithm obtains higher average recognition rate, which shows that the proposed algorithm can well control the data overlap (outlier) and sparse points. The results also show that the proposed algorithm is more discriminative than other algorithms because the membership degree of the fuzzy neighborhood helps to make the nearest neighbor samples closer to each other in the same class, while repel the far neighbor samples of different classes farther.(3)
As shown in Figures2, 4, and 6, under the same experimental conditions, the average recognition rate (%) of the proposed algorithm is better than that of other algorithms. It is further illustrated that the algorithm can obtain the optimal sparse projection discriminant feature because elastic network regression can obtain sparse feature extraction results.(4)
Different from PCA, MMC, and LDA which attempt to preserve the global Euclidean structure, LPP, LLE, SDE, FLGE/MMC, and the proposed algorithm aim to discover the local geometric structure. However, sparse subspace learning methods are superior to the compared methods. The sparse projections learned by the proposed algorithm provide physical interpretation in real-world applications. Therefore, the discriminant knowledge is effectively discovered.Figure 6
The average recognition rates (%) of various algorithms corresponding to various dimensions when five face images of each class are randomly selected as the training set in the AR face database.


## 4.1. The Experiments and Interpretations in the UCI Wine Database

Now, we use a wine database from the UCI machine learning library to further validate and interpret the proposed method. There are 3 classes, 13 features, and 178 instances in the wine database. We select 48 samples in each class, and the basic statistics of the variance, mean, and range of the 13 features in the experiment are shown in Table2.Table 2
The first sparse projection axis and the corresponding sparse load (k = 5) of the proposed method and USSL.
FeaturesMeanVarianceRangeUSSLThe proposed method113.00.73.80022.31.25.10.130.2832.40.081.900419.511.219.40.090.36599.7204.092.00.02062.30.42.90072.01.04.7−0.99−0.9580.40.020.50091.60.33.200105.15.411.700.19111.00.051.200122.60.52.70013746.999166.714020.01040.0008Both the proposed method and USSL sparse projections are suitable for classification. We learn that there are precise physical explanations, which provide us a deeper understanding of the database. That is to say, the second (malic acid), the fourth (ash alkaline), the fifth (magnesium), the seventh (xanthic acid), the tenth (color intensity), and the thirteenth (proline) are the most important and significant distinguishing features. Moreover, the proposed method consistently outperforms USSL within lower dimension.


## 4.2. The Experiments in the ORL Database

There are 40 classes in the ORL face database [43]; each class contains 10 different face images for a total of 400 face images, among which facial expression, posture, and illumination vary. The images allow the face to tilt and rotate 20 degrees. In addition, there are some changes as high as about 10% in this scale. Figure 1 shows ten images of one class in the ORL face database. All images were normalized to 56 × 46 resolution.Figure 1
Ten images of one class in the ORL face database.In this experiment, the performance of PCA, MMC, LDA, LLE, LPP, SDE, FLGE/MMC, and the new proposed method is tested, respectively. First of all,T images (T ranging from 2 to 6) from each class of the image database are randomly selected as the training sample set. Secondly, the remaining 10−T face images of each class are selected as the testing sample set. Finally, we independently run 50 times on T images of each class.In the PCA phase of seven algorithms (MMC, LDA, LPP, LLE, SDE, FLGE/MMC, and the proposed algorithm), 95% of the face image energy is retained for computational effectiveness. Table3 displays the standard deviation, the best average recognition rates (%), and corresponding feature dimension of various algorithms in the ORL face database.Table 3
The standard deviation, the best average recognition rates (%), and corresponding feature dimension of various algorithms in the ORL face database.
MethodsT23456PCA72.25±2.61 (44)81.62±0.65 (44)83.98±1.00 (46)85.41±1.30 (44)87.41±1.98 (46)LDA76.35±1.07 (38)83.40±1.69 (38)84.13±2.03 (38)86.07±1.16 (38)88.60±0.78 (38)MMC75.01±1.77 (40)83.96±0.48 (38)85.37±2.48 (40)87.47±0.85 (38)90.49±1.17 (36)LPP73.52±0.05 (42)80.28±0.78 (44)89.18±1.42 (44)91.13±0.82 (40)92.12±1.21 (38)LLE69.78±0.82 (46)72.66±0.73 (50)76.32±1.14 (50)78.85±1.15 (50)88.44±1.55 (48)SDE77.29±1.20 (28)81.20±0.27 (28)87.89±1.39 (28)89.24±2.41 (26)90.12±0.68 (30)FLGE/MMC79.30±0.83 (46)88.61±1.79 (42)93.53±1.82 (50)95.84±1.23 (48)97.36±1.24 (50)The proposed method91.75±0.45 (26)93.14±1.08 (26)95.14±1.22 (30)96.30±1.24 (26)98.13±1.23 (24)Figure2 shows the curve when four face images in each class are randomly selected as the training set in the ORL face database. The curve shows the average recognition rate (%) of each algorithm corresponding to different dimensions.Figure 2
The average recognition rates (%) of various algorithms corresponding to various dimensions when four face images of each class are randomly selected as the training set in the ORL face database.


## 4.3. The Experiments in the Yale Database

There are 15 classes in the Yale face database [44]; each class contains 11 different face images, a total of 165 face images, among which normal, surprised, happy, sad, sleepy, and winking are varied. Figure 3 shows 11 images of one class in the Yale face database.Figure 3
Eleven images of one class in the Yale face database.In our experiment, all images were normalized to 50 × 40 resolution in the Yale database. It also retains 95% of face image energy as the same as the ORL face database. First of all,T images (T ranging from 2 to 6) from each class of image database are randomly selected as the training sample set. Secondly, the remaining 11−T face images of each class are selected as the testing sample set. Finally, we independently run 50 times on T images of each class. The curve in Figure 4 shows the average recognition rate (%) of various algorithms under different feature dimensions when six face images of each class are randomly selected as training sets in the Yale face database. Table 4 shows the standard deviation, the best average recognition rate (%), and the corresponding feature dimensions of various algorithms in the Yale face database, respectively.Figure 4
The average recognition rates (%) of various algorithms corresponding to various dimensions when six face images of each class are randomly selected as the training set in the Yale face database.Table 4
The standard deviation, best average recognition rates (%), and corresponding feature dimension of various algorithms in the Yale face database.
MethodsT23456PCA77.29±1.20 (38)80.20±1.27 (40)83.99±1.38 (38)84.72±1.24 (40)86.21±0.80 (46)LDA80.45±1.48 (16)84.55±1.06 (16)87.85±0.45 (16)87.20±1.64 (14)88.54±0.82 (14)MMC80.58±0.71 (12)82.84±0.88 (12)85.87±1.12 (12)86.83±0.37 (12)87.79±0.50 (14)LPP80.40±1.80 (24)85.79±0.38 (22)88.00±0.50 (20)89.57±0.70 (18)91.05±1.23 (26)LLE83.55±1.38 (14)83.58±0.59 (6)85.90±0.75 (6)88.37±1.63 (8)88.97±1.56 (8)SDE85.18±0.70 (48)86.48±0.82 (46)87.85±1.20 (46)90.16±1.45 (48)92.47±1.32 (46)FLGE/MMC90.45±0.75 (46)91.48±1.01 (46)93.21±1.24 (44)95.34±1.21 (42)96.33±0.21 (46)The proposed method91.50±0.55 (36)92.58±0.17 (36)95.53±0.86 (38)97.12±0.37 (38)98.75±0.35 (36)


## 4.4. The Experiments in the AR Database

The AR face database [45] has 70 men and 56 women, a total of 126 people. The database consists of more than 4000 color people, including occluded front views, lighting conditions, and facial expressions. In AR database, there are 120 face images, including 65 male and 55 female. These images are divided into two parts (two weeks apart), each part containing 20 color face images. The face part image of each person is normalized to 50 × 40 pixels. And Figure 5 displays an example image of one class in the AR face database.Figure 5
Twenty images of one person in the AR database. The first and second lines of face images were taken at two-week intervals.It also retains 95% of face image energy as the same as ORL and Yale face database. First of all,T images (T ranging from 2 to 6) from each class of the image database are randomly selected as the training sample set. Secondly, the remaining 20−T face images of each class are selected as the testing sample set. Finally, we independently run 10 times on T images of each class. Table 5 shows the standard deviation, the best average recognition rate (%), and the corresponding feature dimensions of various algorithms in the AR face database, respectively.Table 5
The standard deviation, best average recognition rates (%), and corresponding feature dimension of various algorithms in the AR face database.
MethodsT23456PCA66.68 ± 1.11 (145)70.21 ± 1.62 (145)77.60 ± 1.27 (150)79.03 ± 0.81 (150)81.54 ± 1.04 (145)LDA71.5 ± 0.71 (110)75.58 ± 0.76 (115)82.53 ± 0.81 (120)87.12 ± 0.33 (115)87.58 ± 0.75 (120)MMC69.39 ± 1.15 (120)75.71 ± 0.59 (115)82.98 ± 0.62 (115)85.27 ± 0.94 (120)87.86 ± 0.82 (120)LPP72.82 ± 0.81 (85)78.78 ± 1.10 (85)83.70 ± 0.65 (80)88.75 ± 0.31 (85)89.93 ± 0.48 (70)LLE70.40 ± 0.89 (125)74.31 ± 1.16 (145)83.35 ± 1.49 (140)84.78 ± 0.76 (150)86.98 ± 0.40 (145)SDE74.90 ± 0.08 (85)80.10 ± 1.36 (75)84.35 ± 1.36 (80)89.80 ± 0.65 (85)92.50 ± 1.18 (70)FLGE/MMC75.09 ± 1.49 (75)82.42 ± 1.08 (70)88.23 ± 0.8 (75)90.17 ± 1.36 (75)93.38 ± 1.20 (70)The proposed method79.31 ± 1.65 (80)85.79 ± 0.75 (90)90.86 ± 0.82 (95)92.74 ± 0.95 (85)95.82 ± 0.68 (90)Figure2 shows the curve when five face images in each class are randomly selected as the training set in the AR face database. The curve shows the average recognition rate (%) of each algorithm corresponding to different dimensions.


## 4.5. Overall Observations and Discussion

After analyzing the experimental results of the above four databases, the following conclusions can be drawn:(1)
As can be seen from Table1, the first six important characteristics (the second, fourth, fifth, seventh, tenth, and thirteenth characteristics) reflect the data distribution. The last characteristic factor plays a decisive role in the data distribution because of its wide range and large variance.(2)
It can be seen from Tables3–5 that compared with PCA, MMC, LDA, LLE, LPP, SDE, and FLGE/MMC, the proposed algorithm obtains higher average recognition rate, which shows that the proposed algorithm can well control the data overlap (outlier) and sparse points. The results also show that the proposed algorithm is more discriminative than other algorithms because the membership degree of the fuzzy neighborhood helps to make the nearest neighbor samples closer to each other in the same class, while repel the far neighbor samples of different classes farther.(3)
As shown in Figures2, 4, and 6, under the same experimental conditions, the average recognition rate (%) of the proposed algorithm is better than that of other algorithms. It is further illustrated that the algorithm can obtain the optimal sparse projection discriminant feature because elastic network regression can obtain sparse feature extraction results.(4)
Different from PCA, MMC, and LDA which attempt to preserve the global Euclidean structure, LPP, LLE, SDE, FLGE/MMC, and the proposed algorithm aim to discover the local geometric structure. However, sparse subspace learning methods are superior to the compared methods. The sparse projections learned by the proposed algorithm provide physical interpretation in real-world applications. Therefore, the discriminant knowledge is effectively discovered.Figure 6
The average recognition rates (%) of various algorithms corresponding to various dimensions when five face images of each class are randomly selected as the training set in the AR face database.


## 5. Conclusions

This paper proposes a new supervised learning algorithm for feature extraction and face recognition, namely, sparse graph embedding with fuzzy set for image classification. The experimental results show the proposed algorithm is more effective than other techniques. For future work, we will decide optimal parameters such asμ,Kc, andKp and perform more tests on other types of face data. Specifically, for face recognition, the sparse face subspace shows us an intuitive, semantic, and insightful understanding of the feature extraction. In future work, we will extend the SDE algorithm to kernel and tensor forms via the kernel and tensor methods, respectively.



---

*Source: 6638985-2021-01-16.xml*