# Modified Hybrid Discriminant Analysis Methods and Their Applications in Machine Learning

Authors: Liwen Huang
Journal: Discrete Dynamics in Nature and Society
 (2020)

---

## Abstract

This paper presents a new hybrid discriminant analysis method, and this method combines the ideas of linearity and nonlinearity to establish a two-layer discriminant model. The first layer is a linear discriminant model, which is mainly used to determine the distinguishable samples and subsample; the second layer is a nonlinear discriminant model, which is used to determine the subsample type. Numerical experiments on real data sets show that this method performs well compared to other classification algorithms, and its stability is better than the common discriminant models.


---

## Body

## 1. Introduction

Multigroup classification or discrimination is an important problem with application in many fields [1], and Fisher’s linear discriminant analysis (FLDA) [2] is a statistical method for multiclass discrimination. Despite being almost 80 years old, it remains to be one of the most widely used methods in light of its simplicity and effectiveness.At present, there are many alternative methods for classification [3, 4], such as FLDA, Quadratic discriminant analysis, genetic algorithm, artificial neural network approaches, decision trees, and SVM. In fact, using a high number of data sets, it is probable that each classifier will work well in some data sets and not so well in others [5], so no method will be universally superior to other methods [6]. However, as the number of classes increases or the number of variables increases, data sets will become more complex, and there is also a corresponding need for a discriminant model with good stability and applicability. Therefore, it is a good idea to use multiple methods to solve the classification problem in discriminant analysis.In general, when constructing discriminant methods, various assumptions are often made, which may not be appropriate because of the complexity of the actual data. FLDA has no requirement on the distribution of data sets, which attracts the interest of many scholars. Hence, FLDA has developed numerous variations for different purposes since its first publication 80 years ago [7–10]. Recently, various methods, including [11–14], have been developed to combine multiple methods to efficiently process complex data or high-dimensional data [15–18]. However, when the data set is more complex, the characteristics of the data may have different relationships in different parts of the data. To solve this problem, Giles et al. [19] used an iterative denoising method to discover the structure and relationship of the data set. Using the linear and nonlinear ideas, Huang and Su [20] proposed a hierarchical discriminant analysis method (HDAM), but this method was not very effective in finding the characteristics of the data. One-hidden-layer neural network is also a linear and nonlinear classification method, but the classification effect is related to the selection of nonlinear functions. SVM is a good classifier, and its effect is related to the choice of kernel function. Generally, some data sets are sensitive to kernel functions, while others are not. Therefore, in order to solve the discriminant problem of more complex data and improve the stability of the model, this paper tries to establish hybrid discriminant analysis based on HDAM and uses an adaptive method to identify the features of the data set.In the following sections, the paper will discuss the new hybrid discriminant analysis methods, their discriminant criterion, numerical experiments conducted, and its conclusion.


## 2. Modified Hybrid Discriminant Analysis Methods (MHDA)

### 2.1. Modified FLDA Criterion

Suppose there arek classes G1,G2,…,Gk, k>0, where Gi:x1i,x2i,…,xnii. Here, ni is the sample size of the class Gi, i=1,2,…,k. In this paper, assuming that x is an arbitrary given sample with x=x11,x12,…,x1m′.According to the idea of the large between-class difference and the small within-class difference, the goal of FLDA is to find an optimal discriminant vectoru by maximizing the following criterion [21, 22]:(1)Ju=u′Sbuu′Swu.Here,Sb=∑i=1knix¯i−x¯x¯i−x¯′, Sw=∑i=1k∑j=1nix¯ji−x¯ix¯ji−x¯i′, x¯i is the mean of Gi, and x¯ is the mean of all classes.Supposeu is the optimal discriminant vectors obtained by the training samples, then the following linear discriminant function can classify the sample type of each class as much as possible:(2)z=u′x.LetCi be the discriminative threshold between the class Gi and the class Gi+1 and C1<C2<⋯<Ck−1, then the Fisher criterion can be described as follows.For any given samplex, the value of z can be calculated by formula (2), then(1)
Ifz<C1, then x∈G1(2)
IfCi−1<z<Ci, then x∈Gi,i=2,3,…,k−1(3)
Ifz>Ck−1, then x∈GkHowever, depending on the projection direction ofu, there may be overlaps among the samples of different classes, which will lead to misclassification of the samples. In order to improve the classification performance, a two-layer discriminant model is established by using the combination of linear and nonlinear discriminant methods. Its main idea can be described as follows: for the first layer, a modified FLDA is constructed to separate the distinguishable samples from each class, and the rest samples are treated as subsamples; for the second layer, a modified nonlinear discriminant method is established to classify the subsample type.Using formula (2), the projection values of the samples in each class can be calculated. Let ai=minz=u′x,x∈Gi and bi=maxz=u′x,x∈Gi, then interval ai,bi is the discriminant range of the class Gi, i=1,2,…,k. However, for the discriminant ranges of different classes, if there is an intersection, then there will be misclassification samples. Therefore, it is necessary to adjust the discriminant ranges of each class so that these discriminant ranges do not intersect each other.Suppose the new discriminant range of the classGi is denoted by ai′,bi′, then for any given two classes, denoted by Gi and Gj, their discriminant ranges will satisfy the following condition:(3)ai′,bi′∩aj′,bj′=ϕ,1≤i<j≤k.Thus, the new linear discriminant criterion can be described by the following form.For any given samplex, let z=u′x, then,(4)x∈Gi,z∈ai′,bi′,subsample,otherwise.


### 2.2. Modified Nonlinear Discriminant Criterion

Suppose subsamples still havek classes (G1′,G2′,…,Gk′), x¯′i is the mean of Gi′ and ni′ is the number of Gi′, i=1,2,…,k.For a given samplex, if x∈Gi′, then the distance between the sample x and Gi′ is defined by the following form:(5)dx,Gi′=x−x¯′i′x−x¯′i.Letdi be the maximum distance between the samples of Gi′ and x¯′i, then di=max1≤j≤ni′xji−x¯′i′xji−x¯′i. Especially, if Gi′ is a spherical-shaped class, di can be regard as the radius of Gi′.To better distinguish the sample type between two spherical-shaped classes, some results were obtained from the perspective of the inclusion or disjoint of the classes, which can be described as follows [20, 23].LetGp′ be spherical-shaped class or spherical shell-shaped class, and let Gq′ be spherical shell-shaped class and di is the radius of Gi′, i=p,q.(1)
SupposeGp′⊂Gq′ and Gp′∩Gq′=ϕ, namely, the samples of Gp′ are surrounded by the samples of Gq′ and there are no cross samples between Gp′ and Gq′, which indicates that the relationship between the two classes is inclusive. For any given sample x,(i)
Ifx∈Gp′, then dx,Gp′/dp<1 and dx,Gq′/dq<1(ii)
Ifx∈Gq′ and x∉Gp′, then dx,Gp′/dp<1 and dx,Gq′/dq<1(iii)
Ifx∉Gp′ and x∉Gq′, then dx,Gp/dp<1 and dx,Gq/dq<1(2)
SupposeGp′⊄Gq′, Gq′⊄Gp′, and Gp′∩Gq′=ϕ, which indicates that the relationship between the two classes is disjoint. For any given sample x,(i)
Ifx∈Gp′, then dx,Gp′/dp<1 and dx,dx,Gq′<1/dq>1(ii)
Ifx∈Gq′, then dx,Gp′/dp<1 and dx,Gq′/dq<1(iii)
Ifx∉Gq′ and x∉Gp′, then dx,Gp′/dp>1 and dx,Gq′/dq>1The abovementioned two results can be generalized to the case of more than two classes, but this method can only solve the discriminant problem when the relationships among classes are inclusive or disjoint.However, in practical application, many classes may not be spherical classes or spherical shell-shaped classes, and their relationships are not necessarily inclusive or disjoint, which limit the application of this method. A feasible idea is to divide nonspherical class into several spherical-shaped classes [24–27], and the purpose is to find the feature of each class and improve the classification performance of discriminant problem.Suppose the classGp is the original class corresponding to Gp′. In order to better find the feature of Gp′, Gp is divided into kp spherical-shaped classes, denoted by B1p,B2p,…,Bkpp, 1≤kp<np, and cip and dip are the center point and the radius of Bip, where Bip∩Bjp=ϕ, 1≤i≠j≤kp.LetQx,Gp be dx,Gp/dp, for any given sample x, then(6)Qx,Gp=min1≤i≤kpdx,Bipdip=0∼1,ifx∈∪1≤i≤kpBip,>1,otherwize.Thus, the samples ofGp′ can be classified by the features of Gp, and using the principle of neighbors, the nonlinear discriminant criterion can be established as follows.For any given samplex, if Qx,Gi′=min1≤p≤kQx,Gp, then x∈Gi′.


### 2.3. Modified Hybrid Discriminant Analysis Method (MHDA)

From Section2.1 and Section 2.2, MHDA can be briefly described in the following five steps:(1)
Find the optimal discriminant vectoru according to training data(2)
Determine the discriminant range of each class, denoted byai′,bi′, i=1,2,…,k(3)
Classify the sample type of each class from modified FLDA criterion (Section2.1) and determine the subsamples of each class(4)
Decompose each class into several spherical-shaped classes, denoted byB1p,B2p,…,Bkpp, 1≤kp<np, and cip and dip are the center point and the radius of Bip, where Bip∩Bjp=ϕ, 1≤i≠j≤kp(5)
For any given samplex in the subclass, classify its type according to the modified nonlinear discriminant criterion (Section 2.2)HDAM is an improved discriminant method based on FLDA. This method has advantages of FLDA, and it can deal with the discriminant problem of one class surrounded by the other class. Its computational complexity isOk∗n∗m∼Ok∗nlogn∗m. For more extensive application, MHDA is proposed on the basis of HDAM. This method can identify the features of the data set well, but its algorithm runs slower than HDAM. In general, the computational complexity of this method is related to the complexity of the data. If the number of features in the data set is represented as tn, then its complexity is Ok∗n∗m∼Ok∗nlogn∗m∗tn.


## 2.1. Modified FLDA Criterion

Suppose there arek classes G1,G2,…,Gk, k>0, where Gi:x1i,x2i,…,xnii. Here, ni is the sample size of the class Gi, i=1,2,…,k. In this paper, assuming that x is an arbitrary given sample with x=x11,x12,…,x1m′.According to the idea of the large between-class difference and the small within-class difference, the goal of FLDA is to find an optimal discriminant vectoru by maximizing the following criterion [21, 22]:(1)Ju=u′Sbuu′Swu.Here,Sb=∑i=1knix¯i−x¯x¯i−x¯′, Sw=∑i=1k∑j=1nix¯ji−x¯ix¯ji−x¯i′, x¯i is the mean of Gi, and x¯ is the mean of all classes.Supposeu is the optimal discriminant vectors obtained by the training samples, then the following linear discriminant function can classify the sample type of each class as much as possible:(2)z=u′x.LetCi be the discriminative threshold between the class Gi and the class Gi+1 and C1<C2<⋯<Ck−1, then the Fisher criterion can be described as follows.For any given samplex, the value of z can be calculated by formula (2), then(1)
Ifz<C1, then x∈G1(2)
IfCi−1<z<Ci, then x∈Gi,i=2,3,…,k−1(3)
Ifz>Ck−1, then x∈GkHowever, depending on the projection direction ofu, there may be overlaps among the samples of different classes, which will lead to misclassification of the samples. In order to improve the classification performance, a two-layer discriminant model is established by using the combination of linear and nonlinear discriminant methods. Its main idea can be described as follows: for the first layer, a modified FLDA is constructed to separate the distinguishable samples from each class, and the rest samples are treated as subsamples; for the second layer, a modified nonlinear discriminant method is established to classify the subsample type.Using formula (2), the projection values of the samples in each class can be calculated. Let ai=minz=u′x,x∈Gi and bi=maxz=u′x,x∈Gi, then interval ai,bi is the discriminant range of the class Gi, i=1,2,…,k. However, for the discriminant ranges of different classes, if there is an intersection, then there will be misclassification samples. Therefore, it is necessary to adjust the discriminant ranges of each class so that these discriminant ranges do not intersect each other.Suppose the new discriminant range of the classGi is denoted by ai′,bi′, then for any given two classes, denoted by Gi and Gj, their discriminant ranges will satisfy the following condition:(3)ai′,bi′∩aj′,bj′=ϕ,1≤i<j≤k.Thus, the new linear discriminant criterion can be described by the following form.For any given samplex, let z=u′x, then,(4)x∈Gi,z∈ai′,bi′,subsample,otherwise.


## 2.2. Modified Nonlinear Discriminant Criterion

Suppose subsamples still havek classes (G1′,G2′,…,Gk′), x¯′i is the mean of Gi′ and ni′ is the number of Gi′, i=1,2,…,k.For a given samplex, if x∈Gi′, then the distance between the sample x and Gi′ is defined by the following form:(5)dx,Gi′=x−x¯′i′x−x¯′i.Letdi be the maximum distance between the samples of Gi′ and x¯′i, then di=max1≤j≤ni′xji−x¯′i′xji−x¯′i. Especially, if Gi′ is a spherical-shaped class, di can be regard as the radius of Gi′.To better distinguish the sample type between two spherical-shaped classes, some results were obtained from the perspective of the inclusion or disjoint of the classes, which can be described as follows [20, 23].LetGp′ be spherical-shaped class or spherical shell-shaped class, and let Gq′ be spherical shell-shaped class and di is the radius of Gi′, i=p,q.(1)
SupposeGp′⊂Gq′ and Gp′∩Gq′=ϕ, namely, the samples of Gp′ are surrounded by the samples of Gq′ and there are no cross samples between Gp′ and Gq′, which indicates that the relationship between the two classes is inclusive. For any given sample x,(i)
Ifx∈Gp′, then dx,Gp′/dp<1 and dx,Gq′/dq<1(ii)
Ifx∈Gq′ and x∉Gp′, then dx,Gp′/dp<1 and dx,Gq′/dq<1(iii)
Ifx∉Gp′ and x∉Gq′, then dx,Gp/dp<1 and dx,Gq/dq<1(2)
SupposeGp′⊄Gq′, Gq′⊄Gp′, and Gp′∩Gq′=ϕ, which indicates that the relationship between the two classes is disjoint. For any given sample x,(i)
Ifx∈Gp′, then dx,Gp′/dp<1 and dx,dx,Gq′<1/dq>1(ii)
Ifx∈Gq′, then dx,Gp′/dp<1 and dx,Gq′/dq<1(iii)
Ifx∉Gq′ and x∉Gp′, then dx,Gp′/dp>1 and dx,Gq′/dq>1The abovementioned two results can be generalized to the case of more than two classes, but this method can only solve the discriminant problem when the relationships among classes are inclusive or disjoint.However, in practical application, many classes may not be spherical classes or spherical shell-shaped classes, and their relationships are not necessarily inclusive or disjoint, which limit the application of this method. A feasible idea is to divide nonspherical class into several spherical-shaped classes [24–27], and the purpose is to find the feature of each class and improve the classification performance of discriminant problem.Suppose the classGp is the original class corresponding to Gp′. In order to better find the feature of Gp′, Gp is divided into kp spherical-shaped classes, denoted by B1p,B2p,…,Bkpp, 1≤kp<np, and cip and dip are the center point and the radius of Bip, where Bip∩Bjp=ϕ, 1≤i≠j≤kp.LetQx,Gp be dx,Gp/dp, for any given sample x, then(6)Qx,Gp=min1≤i≤kpdx,Bipdip=0∼1,ifx∈∪1≤i≤kpBip,>1,otherwize.Thus, the samples ofGp′ can be classified by the features of Gp, and using the principle of neighbors, the nonlinear discriminant criterion can be established as follows.For any given samplex, if Qx,Gi′=min1≤p≤kQx,Gp, then x∈Gi′.


## 2.3. Modified Hybrid Discriminant Analysis Method (MHDA)

From Section2.1 and Section 2.2, MHDA can be briefly described in the following five steps:(1)
Find the optimal discriminant vectoru according to training data(2)
Determine the discriminant range of each class, denoted byai′,bi′, i=1,2,…,k(3)
Classify the sample type of each class from modified FLDA criterion (Section2.1) and determine the subsamples of each class(4)
Decompose each class into several spherical-shaped classes, denoted byB1p,B2p,…,Bkpp, 1≤kp<np, and cip and dip are the center point and the radius of Bip, where Bip∩Bjp=ϕ, 1≤i≠j≤kp(5)
For any given samplex in the subclass, classify its type according to the modified nonlinear discriminant criterion (Section 2.2)HDAM is an improved discriminant method based on FLDA. This method has advantages of FLDA, and it can deal with the discriminant problem of one class surrounded by the other class. Its computational complexity isOk∗n∗m∼Ok∗nlogn∗m. For more extensive application, MHDA is proposed on the basis of HDAM. This method can identify the features of the data set well, but its algorithm runs slower than HDAM. In general, the computational complexity of this method is related to the complexity of the data. If the number of features in the data set is represented as tn, then its complexity is Ok∗n∗m∼Ok∗nlogn∗m∗tn.


## 3. Numerical Examples

To demonstrate the improvements that can be achieved by this method of the paper, nine data sets are derived from UCI Machine Learning Repository [28]; these data sets are Abalone Data set, Balance Scale Data Set, Banknote Authentication Data Set, Breast Tissue Data Set, Cryotherapy Data Set, Iris Data Set, Vehicle Silhouettes Data Set, Vertebral2c Data Set, and Vertebral3c Data Set, respectively, and their basic information is shown in Table 1.Table 1
Information of different data sets.
UCI data set
Number
Variable
Class
Abalone
4177
8
3
Balance scale
625
4
3
Banknote authentication
1372
4
2
Breast tissue
106
9
6
Cryotherapy
90
6
2
Iris
150
4
3
Vehicle silhouettes
846
18
4
Vertebral2c
310
6
2
Vertebral3c
310
6
3Then, for the nine data sets above, the discriminant model is established by FLDA [29], SDAM [23], Bayes Discriminant analysis method (BDAM) [29], HDAM [20], SVM-Kernel (SVM-Kernel method is derived from SVM-KM-matlab, and learning parameters are set as follows: c = 1000, lambda = 1e − 7, kerneloption = 2, and kernel = “poly”, verbose = 1), Ensembles (Ensembles method is derived from Matlab 18, and parameters are set to AdaboostM1 or AdaboostM2) and MHDA, respectively, and their results are given in Table 2.Table 2
Accuracy ratio of different methods.
UCI data set
Accuracy ratio (%)
FLDA
SDAM
BDAM
HDAM
SVM-Kernel
Ensembles
MHDA
Abalone
52.89
49.03
54.32
48.86
N
53.82
79.79
Balance scale
68.64
52.00
69.28
79.84
99.36
88.16
93.12
Banknote
97.67
75.44
97.67
99.20
100.00
100.00
99.78
Breast tissue
52.83
46.23
72.64
66.98
33.96
57.55
78.30
Cryotherapy
90.00
65.56
90.00
67.78
56.67
100.00
96.67
Iris
98.67
93.33
98.00
98.00
99.30
96.67
98.67
Vehicle silhouettes
63.71
31.00
79.67
30.73
25.65
53.31
77.31
Vertebral2c
80.65
76.77
80.65
76.77
67.74
91.94
94.52
Vertebral3c
68.39
76.45
80.97
73.55
48.39
78.06
93.55
Note. In Table 2, N denotes the exception of the corresponding method.Generally, FLDA is suitable for the problem of the large between-class difference and the small within-class difference. SDAM and HDAM can achieve good effect for the discriminant problem of spherical-shaped classes or classes with inclusive relations. BDAM has good classification performance on the discriminant problem of multivariate normality and equal class covariance matrix. Since the real-world data are usually more complex, these four methods do not always work well. Results given in Table2 also illustrate this point and indicate the robustness of these four methods which needs to be improved. Furthermore, from Table 2, although SVM-Kernel and Ensembles are better than MHDA on some data sets, their stability and overall effect are inferior to MHDA.MHDA is an extension of HDAM, and from Table2, its classification accuracy ratio is superior to HDAM. However, the results in Table 3 show that the run time of MHDA is generally longer than that of HDAM.Table 3
Run time of HDAM and MHDA.
UCI data set
Run time (s)
HDAM
MHDA
Abalone
1.6853
44.2816
Balance scale
0.0384
0.6347
Banknote
0.0380
0.4328
Breast tissue
0.0121
0.0839
Cryotherapy
0.0264
0.0525
Iris
0.0199
0.0435
Vehicle silhouettes
0.1583
0.7580
Vertebral2c
0.0155
0.2686
Vertebral3c
0.0344
0.1664Numerical examples indicate, in some data sets, MHDA does not achieve the best results, but it still has a high accuracy ratio. However, it can be seen from Tables1 and 2, when the number of classes in multiple data sets is equal, such as Abalone, Balance Scale, and Vertebral3c, the accuracy ratio tends to decrease as the number of variables increases; when the number of variables in multiple data sets is equal, for example, Vertebral2c and Vertebral3c, or Banknote Authentication and Iris, the accuracy ratio tends to decrease as the number of classes increases; when the number of classes and the number of variables in multiple data sets are all equal, such as Balance Scale and Iris, or Cryotherapy and Vertebral2c, the accuracy ratio is usually related to the complexity of the data set. The run time of MHDA is not necessarily related to the number of variables and classes, but it is related to the size of samples, the number of variables, the number of classes, and the complexity of the data set. From the results of the data sets of Abalone, Breast Tissue, and Vehicle Silhouettes, as the number of variables increases or the number of classes increases, the corresponding data sets tends to become more complex, and the performance of MHDA tends to decline. Thus, for the data set of higher dimensions or more classes, the next goal is to improve the classification performance through the variable selection or dimensionality reduction.


## 4. Conclusions

In this paper, MHDA is proposed based on HDAM, and it combines the ideas of linearity and nonlinearity to establish a two-layer discriminant model. HDAM can better solve the discriminant problem that the relationships among classes are inclusive or disjoint, but the real-world data are usually more complex, which limits its wide application. MHDA overcomes these shortcomings and improves the accuracy ratio and robustness.Numerical experiments show that MHDA works well for the real data sets, and its accuracy ratio and stability are better than some common discriminant methods. However, for the data set of higher dimensions or more classes, the effect achieved by MHDA needs to be further improved. In the future, the method will need to be further modified so that it can be better suit for the discrimination problem of such data sets.



---

*Source: 1512391-2020-02-24.xml*