# Comparing the Linear and Quadratic Discriminant Analysis of Diabetes Disease Classification Based on Data Multicollinearity

Authors: Autcha Araveeporn
Journal: International Journal of Mathematics and Mathematical Sciences
 (2022)

---

## Abstract

Linear and quadratic discriminant analysis are two fundamental classification methods used in statistical learning. Moments (MM), maximum likelihood (ML), minimum volume ellipsoids (MVE), and t-distribution methods are used to estimate the parameter of independent variables on the multivariate normal distribution in order to classify binary dependent variables. The MM and ML methods are popular and effective methods that approximate the distribution parameter and use observed data. However, the MVE and t-distribution methods focus on the resampling algorithm, a reliable tool for high resistance. This paper starts by explaining the concepts of linear and quadratic discriminant analysis and then presents the four other methods used to create the decision boundary. Our simulation study generated the independent variables by setting the coefficient correlation via multivariate normal distribution or multicollinearity, often through basic logistic regression used to construct the binary dependent variable. For application to Pima Indian diabetic dataset, we expressed the classification of diabetes as the dependent variable and used a dataset of eight independent variables. This paper aimed to determine the highest average percentage of accuracy. Our results showed that the MM and ML methods successfully used large independent variables for linear discriminant analysis (LDA). However, the t-distribution method of quadratic discriminant analysis (QDA) performed better when using small independent variables.


---

## Body

## 1. Introduction

Logistic regression is a standard statistical algorithm used to classify binary dependent Logistic regression is a standard statistical algorithm to classify binary dependent variables based on independent variables. Nevertheless, the critical assumption of logistic regression is that there is a linear decision surface between the dependent and independent variables. Furthermore, logistic regression requires an average amount of or no multicollinearity problems between independent variables. Linearly separable data and multicollinearity in data are often found in real-world situations. It is hard to determine complicated relationships using logistic regression. More powerful and concise algorithms such as discriminant analysis [1] can easily exceed the performance of this algorithm.Discriminant analysis is used for classification, dimension, and data visualization. The classification was explored in a study of discriminant analysis [2] that has been used in many classification problems [3, 4]. When performing discriminant analysis, users can discuss classification methods in which two or more groups and one or more independent variables are placed into one of the measured characteristics. Medical scientists investigate how groups (characterized by blood pressure, blood glucose levels, and age) differ across independent variables. One study used discriminant analysis to determine which patients had previously suffered a heart attack [5] to classify whether the patient would survive based on other variables. Two discriminant analyses are interesting: linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA).LDA can be interpreted from decision boundaries at two points. The first is a probabilistic interpretation, and the second is an interpretation of Fisher [6]. The first interpretation helps consider the assumptions of LDA. The second interpretation provides a better understanding of how LDA performs classification. Fisher [7] introduced an LDA that identified a linear combination of the independent variables with a maximum class separation ratio. LDA’s ability to successfully distinguish between tumor classes [8] using gene expression data was an essential feature of a new approach to cancer classification. LDA assumes that the variance between all classes is equal, and the decision boundary is calculated in a linear function. In other words, when each class has an individual covariance matrix, this is QDA, and the decision boundary shows in the form of a quadratic function. The LDA and QDA [9] should be recommended when using NIR data to classify an ill-posed problem. The principal component analysis [10] was applied with LDA and QDA for discriminants between healthy control and cancer samples based on MS data sets. The seven classification methods [11] were tested with binary logistic, probit, and cumulative probit regression, LDA, QDA, artificial neural networks, and naïve Bayes classification, to examine skeletal sex estimation. Consequently, LDA may be more preferred in skeletal sex estimation than other methods. LDA and QDA proposed flow regime identification [12] to combine responses from a non-intrusive optical sensor for air and water’s vertical upward gas-liquid flow.The LDA and QDA methods use the multivariate normal distribution of independent variables as a classification rule. The parameter of prior probability, mean, and covariance matrix of each class also create the discriminant function for the boundary of classes. The two classes are assumed to have the normal distribution, the most common and default distribution in real-world applications. Therefore, if the normal distribution is considered for the two classes where the covariance matrices are assumed to be equal, the decision boundary of classification is in the form of a linear discriminant function. When the covariance matrices are assumed to be unequal, the decision boundary of classification is in the form of a quadratic discriminant function.This study aimed to investigate the parameter estimations for LDA and QDA. We proposed commonly-used methods, namely the moments (MM) and maximum likelihood (ML) methods, and the iterative algorithm, namely the minimum value ellipsoid (MVE) and t-distribution methods, to determine which conditional distribution was from the multivariate normal distribution.Traditionally, LDA and QDA are estimated using the first and the second theoretical moments, also called the MM method introduced by Bowman and Shenton [13] in their concept of the central limit theorem. This method estimated population parameters that expressed population moments as the parameter of interest in terms of the functions. These functions were then set equal to the sample moments, and the number of equations was the same as the number of parameters. The equations were then used to estimate the parameters of interest before the estimators were solved to approximate these parameters.The ML method [14] was a technique that was widely used for estimating probability distribution function parameters based on the observed data. The parameters were estimated by maximizing the likelihood function of observation. The logarithm was then based on the likelihood function. Taking the derivative with respect to a parameter was straightforward, and when set to zero, the ML estimators produced a pleasant result.Rousseeuw [15, 16] studied the resampling algorithm using the minimum volume ellipsoid method. This method generated a robust estimation of multivariate location and scale. It estimated a low bias and expedient for outlier inspection via multivariate data, often through the benefit of robust distances on the MVE estimate. The MVE properties included breakdown value, affine equivariance, and efficiency.The suitable t-distribution method [17] used the statistical modeling finite mixtures model to classify multivariate data sets. Its advantages were that it was the most comfortable method for modeling, and it obtained increasing determination. Multivariate normal data had been used because of their easy computation. Modeling that uses a mixture of t-distribution methods can be fitted using an ML iteration [18] expectation-maximization (EM) algorithm.The debate on using machine learning or discriminant analysis (LDA and QDA) has been one of the critical questions in classification. Generally speaking, the discriminative classifiers model directly depends on the probability distribution function and thus needs fewer parameters; this alleviates the difficulty of parameter estimation problems. Machine learning is concerned with the algorithms that learn how to assign a class label to an example from the problem domain. On the other hand, because class probability distributions are known, LDA and QDA have unique advantages in dealing with multicollinearity data. The ML, MM, MVE, and t-distribution methods are approximate mean and covariance from the multivariate normal distribution that plays important role in classification.This study compares the MM, ML, MVE, and t-distribution methods depending on the LDA and QDA. These methods were used to classify the binary dependent variables related to the multicollinearity of independent variables. Their percentage of accuracy determined the performance of the four methods. Additionally, we clarified some of the simulation study’s theoretical concepts and real data.


## 2. The Concept of Linear and Quadratic Discriminant Methods

The probability density function for multivariate normal distribution, orx∼Nμ,∑, is written as:(1)fx|μ,Σ=12πp∑exp−x−μT∑−1x−μ2,where x=x1,x2,...,xp is the independent variable, μ=μ1,μ2,...,μp is the mean of the independent variables, and Σ is the covariance matrix.


### 2.1. Linear Doscriminant Analysis (LDA)

LDA [19] assumes that the binary classification has the equal covariance matrix Σ1=Σ2=Σ. Therefore, the equation (1) becomes:(2)12πp∑exp−x−μ1T∑−1x−μ12π1=12πp∑exp−x−μ2T∑−1x−μ22π2,exp−x−μ1T∑−1x−μ12π1=exp−x−μ2T∑−1x−μ22π2,where π1 and π2 are the prior probability of the two classes and μ1 and μ2 are the mean of the two classes.Using the natural logarithm from two sides of the equation, we get the simplified term:(3)−12x−μ1T∑−1x−μ1+lnπ1=−12x−μ2T∑−1x−μ2+lnπ2,−12xT∑−1x−12μ1T∑−1μ1+μ1T∑−1x+lnπ1=−12xT∑−1x−12μ2T∑−1μ2+μ2T∑−1x+lnπ2,where equation (3) is xT∑−1μ1=μ1T∑−1x, and when each side is multiplied by 2, we get:(4)2∑−1μ2−μ1Tx+μ2−μ1T∑−1μ2−μ1+2lnπ2π1=0.Equation (4) shows the form of a linear term. LDA discriminant function the two classes as:(5)δx=2∑−1μ2−μ1Tx+μ2−μ1T∑−1μ2−μ1+2lnπ2π1=0.The class of instancex is defined by:(6)c^x=1,ifδx<0,2,ifδx>0.The LDA Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Compute the mean of each classμ1 and μ2.(3)
Calculate the prior probability of each classπ1 and π2.(4)
Compute the covariance matrix for each classΣ1 and Σ2.(5)
Approximate the pooled covariance matrix asΣ^=n1−1∑^1+n2−1∑^2/n−2,, n=n1+n2.(6)
Calculate the LDA discriminant function as in equation (5).(7)
Assign the class label as in as in equation (6).


### 2.2. Quadratic Doscriminant Analysis (QDA)

QDA [19] for binary classification is defined as the unequal covariance matrix Σ1≠Σ2. Therefore, equation (2) can be adjusted in term of unequal covariance as:(7)12πp∑1exp−x−μ1T∑1−1x−μ12π1=12πp∑2exp−x−μ2T∑2−1x−μ22π2.The natural logarithm taken from equation (7) is: (8)−p2ln2π−12ln∑1−12x−μ1T∑1−1x−μ1+lnπ1=−p2ln2π−12ln∑2−12x−μ2T∑2−1x−μ2+lnπ2,−12ln∑1−12xT∑1−1x−12μ1T∑1−1μ1+μ1T∑1−1x+lnπ1=−12ln∑2−12xT∑2−1x−12μ2T∑2−1μ2+μ2T∑2−1x+lnπ2.We multiplied the sides of the previous equation by 2 and got:(9)xT∑1−∑2−1x+2∑2−1μ2−∑1−1μ1Tx+μ1t∑1−1μ1−μ2t∑2−1μ2+ln∑1∑2+2lnπ2π1=0.To obtain Equation (8) in the quadratic form xTAx+bTx+c=0, we brought the expression to the QDA discriminant function as:(10)δx≔xT∑1−∑2−1x+2∑2−1μ2−∑1−1μ1Tx+μ1t∑1−1μ1−μ2t∑2−1μ2,+ln∑1∑2+2lnπ2π1=0.The classification term is shown in equation (6).The QDA Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Compute the mean of each classμ1 and μ2.(3)
Calculate the prior probability of each classπ1 and π2.(4)
Compute the covariance matrix for each classΣ1 and Σ2.(5)
Calculate the QDA discriminant function as in equation (10).(6)
Assign the class label as in as in equation (6).


## 2.1. Linear Doscriminant Analysis (LDA)

LDA [19] assumes that the binary classification has the equal covariance matrix Σ1=Σ2=Σ. Therefore, the equation (1) becomes:(2)12πp∑exp−x−μ1T∑−1x−μ12π1=12πp∑exp−x−μ2T∑−1x−μ22π2,exp−x−μ1T∑−1x−μ12π1=exp−x−μ2T∑−1x−μ22π2,where π1 and π2 are the prior probability of the two classes and μ1 and μ2 are the mean of the two classes.Using the natural logarithm from two sides of the equation, we get the simplified term:(3)−12x−μ1T∑−1x−μ1+lnπ1=−12x−μ2T∑−1x−μ2+lnπ2,−12xT∑−1x−12μ1T∑−1μ1+μ1T∑−1x+lnπ1=−12xT∑−1x−12μ2T∑−1μ2+μ2T∑−1x+lnπ2,where equation (3) is xT∑−1μ1=μ1T∑−1x, and when each side is multiplied by 2, we get:(4)2∑−1μ2−μ1Tx+μ2−μ1T∑−1μ2−μ1+2lnπ2π1=0.Equation (4) shows the form of a linear term. LDA discriminant function the two classes as:(5)δx=2∑−1μ2−μ1Tx+μ2−μ1T∑−1μ2−μ1+2lnπ2π1=0.The class of instancex is defined by:(6)c^x=1,ifδx<0,2,ifδx>0.The LDA Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Compute the mean of each classμ1 and μ2.(3)
Calculate the prior probability of each classπ1 and π2.(4)
Compute the covariance matrix for each classΣ1 and Σ2.(5)
Approximate the pooled covariance matrix asΣ^=n1−1∑^1+n2−1∑^2/n−2,, n=n1+n2.(6)
Calculate the LDA discriminant function as in equation (5).(7)
Assign the class label as in as in equation (6).


## 2.2. Quadratic Doscriminant Analysis (QDA)

QDA [19] for binary classification is defined as the unequal covariance matrix Σ1≠Σ2. Therefore, equation (2) can be adjusted in term of unequal covariance as:(7)12πp∑1exp−x−μ1T∑1−1x−μ12π1=12πp∑2exp−x−μ2T∑2−1x−μ22π2.The natural logarithm taken from equation (7) is: (8)−p2ln2π−12ln∑1−12x−μ1T∑1−1x−μ1+lnπ1=−p2ln2π−12ln∑2−12x−μ2T∑2−1x−μ2+lnπ2,−12ln∑1−12xT∑1−1x−12μ1T∑1−1μ1+μ1T∑1−1x+lnπ1=−12ln∑2−12xT∑2−1x−12μ2T∑2−1μ2+μ2T∑2−1x+lnπ2.We multiplied the sides of the previous equation by 2 and got:(9)xT∑1−∑2−1x+2∑2−1μ2−∑1−1μ1Tx+μ1t∑1−1μ1−μ2t∑2−1μ2+ln∑1∑2+2lnπ2π1=0.To obtain Equation (8) in the quadratic form xTAx+bTx+c=0, we brought the expression to the QDA discriminant function as:(10)δx≔xT∑1−∑2−1x+2∑2−1μ2−∑1−1μ1Tx+μ1t∑1−1μ1−μ2t∑2−1μ2,+ln∑1∑2+2lnπ2π1=0.The classification term is shown in equation (6).The QDA Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Compute the mean of each classμ1 and μ2.(3)
Calculate the prior probability of each classπ1 and π2.(4)
Compute the covariance matrix for each classΣ1 and Σ2.(5)
Calculate the QDA discriminant function as in equation (10).(6)
Assign the class label as in as in equation (6).


## 3. The Estimation Methods

The classification of these methods focused on the probability density function for multivariate normal distribution from equation (1), x∼Nμ,∑, and is written as:(11)fx|μ,Σ=12πp∑exp−12x−μT∑−1x−μ.This section estimated parameters from multivariate normal distribution across the four methods.


### 3.1. Moments (MM) Method

The MM method relates the equivalent of sample and theoretical moments. Sample moments are denoted whereEXk is the kth moment of the distribution for k=1,2,..., and Mk=1/n∑i=1nXik is the kth sample moment for k=1,2,....For multivariate normal distribution, the first sample moment approximates the origin by.M1=1/n∑i=1nxi=x¯ to the first theoretical moment EX. Then, the second sample moment approximates the origin using M2=1/n∑i=1nxi2=1/n∑i=1nxixiT to the second theoretical moment EX2. The first and the second theoretical moments approximate the origin by EXi=μ and. EXi2=Σ+μμT.The first theoretical moments approximate the origin with the sample moment and the moment estimator ofμ is:(12)EXi=μ^mm=1n∑i=1nxi.The second theoretical moment approximates the origin with the sample moment and we obtain:(13)EXi2=Σ+μμT=1n∑i=1nxi2=1n∑i=1nxixiT,where μ^mm=1/n∑i=1nxi, and the moments estimator of Σ is approximated as:(14)Σ^MM=1n∑i=1nxixiT−μ^MMμ^MMT=1n∑i=1nxi−μ^MMxi−μ^MMT.


### 3.2. Maximum Likelihood (ML) Method

From the probability density function, the likelihood function is created by(15)Lμ,Σ=2π−np/2∑−n/2exp−12∑i=1nxi−μT∑−1xi−μ,where x=xi1,xi2,...,xip and μ=μ1,μ2.The likelihood function is calculated using the logarithm to the log likelihood function:(16)lμ,Σ=−np2log2π−n2log∑−12∑i=1nxi−μT∑−1xi−μ.Taking the derivative with respect toμ is straightforward:(17)∂lμ,Σ∂μ=∑i=1nxi−μT∑−1,and setting to zero we obtain a pleasant result:(18)μ^ML=1n∑i=1nxi.The ML estimator is the mean in the sample mean. The covariance matrix is approximated by taking the derivative with respect to the matrix fromΣ−1 using equation (16).(19)∂lμ,Σ∂∑−1=n2Σ−12∑i=1nxi−μTxi−μ.Finally, setting to zero yields the ML estimator:(20)Σ^ML=1n∑i=1nxi−μ^MLxi−μ^MLT.It can be concluded that the ML and the MM estimators used the same equation and values.The MM and ML Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Select the sample sizes of each classn1 and n2.(3)
Calculate the mean of each classμ^1=∑i=1n1xi and μ^2=1/n∑i=1n2xi(4)
Compute the covariance matrix for each classΣ^1=1/n∑i=1n1xi−μ^1xi−μ^1T and Σ^2=1/n∑i=1n2xi−μ^2xi−μ^2T


### 3.3. Minimum Value Ellipsoid(MVE) Method

This method [20] depends on the MVE, which covers the least point h of the n observations. The multivariate data is defined as x=x1,x2,...,xp with independent variables p and xi=xi1,...,xipT;i=1,2,...,n. Computing the MVE for a set of data x requires the examination of all nh ellipsoids that contain the observed h=n+p+1/2 of x for the smallest ellipsoid volume. The MVE algorithms defined ellipsoids by including the subsets of observed p+1 on data x. For each subset of size p+1, indicated by J=i1,...,ip+1⊂1,...,n, the sample mean and sample covariance matrix were computed by:(21)μ^MVEJ=1p+1∑j=1p+1xij,Σ^MVEJ=1p+1∑j=1p+1xij−x¯Jxij−x¯JT.The covariance matrixΣJ is a nonsingular covariance matrix when the subset of p+1 is in general rank. If the subset of p+1 was not in general rank, then the observed data from x were included until a subset with the sample of nonsingular covariance matrix was achieved.The MVE Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Compute the standard choiceh=n+p+1/2.(3)
Require examination of allnh ellipsoids.(4)
Search to ellipsoids determined by subsets consisting ofp+1 obsavations.(5)
Compute meanμ^MVEJ and covariance Σ^MVEJ.


### 3.4. The T-distribution Method

The modeling of a mixture of t-distributions [17] is considered a robust estimator that fits the data set with the ML via the expectation-maximization (EM) algorithm [18]. The t-distribution proposes a longer-tailed distribution that is closed to the normal distribution for large sample sizes.The data set is denoted byx=x1,x2,...,xp with p independent variables from sample sizes n. In a normal mixture model based on these data, each data set is assumed to be a comprehension of the random variables of dimensional vector p with the g-component normal mixture probability density function on data set x:(22)fx;Ψ=∑j=1gπjϕx;μj,Σj,where the proportions πj are nonnegative and all proportions sum to one. Then(23)ϕx;μj,Σj=2π−p/2∑−1/2×exp−12x−μjT∑j−1x−μj,denotes thep-variate multivariate normal probability density function with mean μi and covariance matrix ∑jj=1,…g. Here Ψ=π1,...,πg−1,θTT where θ is composed of the components of the μj and the distinct components of the Σjj=1,...,g.Thet mixture model depending on the normal distribution is obtained with the location parameter μ, positive covariance matrix Σ, and ν degrees of freedom,(24)fx;μ,Σ,ν=Γν+p/2Σ−1/2πν1/2Γν/21+δx,μ,Σ/ν1/2v+p,where δx,μ,Σ=x−μTΣ−1x−μ.The implementation of the EM algorithm for ML estimation in the case of a single component t-distribution has been discussed in MaLachlan et al. [21]. For the EM algorithm at the iterations k+1 th, πk+1,θk+1 , and νk+1 can be computed independently of each other. To compute the weight sample mean and sample covariance matrix, we used:(25)μ^jk+1=∑i=1nτijkuijkxi∑i=1nτijkuijk,Σ^jk+1=∑i=1nτijkuijkxi−μjk+1xi−μjk+1T∑i=1nτijk.Where uijk=νjk+p/νjk+δxi,μjk;Σjk and τijk=πjkfxi;μjk+1,νjk+1/fxi;Ψk+1.It can be seen thatμ^ik+1 and Σ^jk+1 were selected after weighted least squares estimation. The E-step drew the weights uijk while the M-step successfully chose μ^ik+1 and Σ^jk+1 using the weighted least squares method, which we called the EM algorithm.The t-distribution Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Assumex=x1,x2,...,xp to be the g-component normal mixture probability density function.(3)
Obtain the parameter mean and covariance to t-distribution.(4)
Implement EM algorithm at the iterationsk+1 th, πk+1,θk+1 , and νk+1.(5)
Compute the weight sample mean and sample covariance matrix asμ^jk+1 and Σ^jk+1.


## 3.1. Moments (MM) Method

The MM method relates the equivalent of sample and theoretical moments. Sample moments are denoted whereEXk is the kth moment of the distribution for k=1,2,..., and Mk=1/n∑i=1nXik is the kth sample moment for k=1,2,....For multivariate normal distribution, the first sample moment approximates the origin by.M1=1/n∑i=1nxi=x¯ to the first theoretical moment EX. Then, the second sample moment approximates the origin using M2=1/n∑i=1nxi2=1/n∑i=1nxixiT to the second theoretical moment EX2. The first and the second theoretical moments approximate the origin by EXi=μ and. EXi2=Σ+μμT.The first theoretical moments approximate the origin with the sample moment and the moment estimator ofμ is:(12)EXi=μ^mm=1n∑i=1nxi.The second theoretical moment approximates the origin with the sample moment and we obtain:(13)EXi2=Σ+μμT=1n∑i=1nxi2=1n∑i=1nxixiT,where μ^mm=1/n∑i=1nxi, and the moments estimator of Σ is approximated as:(14)Σ^MM=1n∑i=1nxixiT−μ^MMμ^MMT=1n∑i=1nxi−μ^MMxi−μ^MMT.


## 3.2. Maximum Likelihood (ML) Method

From the probability density function, the likelihood function is created by(15)Lμ,Σ=2π−np/2∑−n/2exp−12∑i=1nxi−μT∑−1xi−μ,where x=xi1,xi2,...,xip and μ=μ1,μ2.The likelihood function is calculated using the logarithm to the log likelihood function:(16)lμ,Σ=−np2log2π−n2log∑−12∑i=1nxi−μT∑−1xi−μ.Taking the derivative with respect toμ is straightforward:(17)∂lμ,Σ∂μ=∑i=1nxi−μT∑−1,and setting to zero we obtain a pleasant result:(18)μ^ML=1n∑i=1nxi.The ML estimator is the mean in the sample mean. The covariance matrix is approximated by taking the derivative with respect to the matrix fromΣ−1 using equation (16).(19)∂lμ,Σ∂∑−1=n2Σ−12∑i=1nxi−μTxi−μ.Finally, setting to zero yields the ML estimator:(20)Σ^ML=1n∑i=1nxi−μ^MLxi−μ^MLT.It can be concluded that the ML and the MM estimators used the same equation and values.The MM and ML Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Select the sample sizes of each classn1 and n2.(3)
Calculate the mean of each classμ^1=∑i=1n1xi and μ^2=1/n∑i=1n2xi(4)
Compute the covariance matrix for each classΣ^1=1/n∑i=1n1xi−μ^1xi−μ^1T and Σ^2=1/n∑i=1n2xi−μ^2xi−μ^2T


## 3.3. Minimum Value Ellipsoid(MVE) Method

This method [20] depends on the MVE, which covers the least point h of the n observations. The multivariate data is defined as x=x1,x2,...,xp with independent variables p and xi=xi1,...,xipT;i=1,2,...,n. Computing the MVE for a set of data x requires the examination of all nh ellipsoids that contain the observed h=n+p+1/2 of x for the smallest ellipsoid volume. The MVE algorithms defined ellipsoids by including the subsets of observed p+1 on data x. For each subset of size p+1, indicated by J=i1,...,ip+1⊂1,...,n, the sample mean and sample covariance matrix were computed by:(21)μ^MVEJ=1p+1∑j=1p+1xij,Σ^MVEJ=1p+1∑j=1p+1xij−x¯Jxij−x¯JT.The covariance matrixΣJ is a nonsingular covariance matrix when the subset of p+1 is in general rank. If the subset of p+1 was not in general rank, then the observed data from x were included until a subset with the sample of nonsingular covariance matrix was achieved.The MVE Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Compute the standard choiceh=n+p+1/2.(3)
Require examination of allnh ellipsoids.(4)
Search to ellipsoids determined by subsets consisting ofp+1 obsavations.(5)
Compute meanμ^MVEJ and covariance Σ^MVEJ.


## 3.4. The T-distribution Method

The modeling of a mixture of t-distributions [17] is considered a robust estimator that fits the data set with the ML via the expectation-maximization (EM) algorithm [18]. The t-distribution proposes a longer-tailed distribution that is closed to the normal distribution for large sample sizes.The data set is denoted byx=x1,x2,...,xp with p independent variables from sample sizes n. In a normal mixture model based on these data, each data set is assumed to be a comprehension of the random variables of dimensional vector p with the g-component normal mixture probability density function on data set x:(22)fx;Ψ=∑j=1gπjϕx;μj,Σj,where the proportions πj are nonnegative and all proportions sum to one. Then(23)ϕx;μj,Σj=2π−p/2∑−1/2×exp−12x−μjT∑j−1x−μj,denotes thep-variate multivariate normal probability density function with mean μi and covariance matrix ∑jj=1,…g. Here Ψ=π1,...,πg−1,θTT where θ is composed of the components of the μj and the distinct components of the Σjj=1,...,g.Thet mixture model depending on the normal distribution is obtained with the location parameter μ, positive covariance matrix Σ, and ν degrees of freedom,(24)fx;μ,Σ,ν=Γν+p/2Σ−1/2πν1/2Γν/21+δx,μ,Σ/ν1/2v+p,where δx,μ,Σ=x−μTΣ−1x−μ.The implementation of the EM algorithm for ML estimation in the case of a single component t-distribution has been discussed in MaLachlan et al. [21]. For the EM algorithm at the iterations k+1 th, πk+1,θk+1 , and νk+1 can be computed independently of each other. To compute the weight sample mean and sample covariance matrix, we used:(25)μ^jk+1=∑i=1nτijkuijkxi∑i=1nτijkuijk,Σ^jk+1=∑i=1nτijkuijkxi−μjk+1xi−μjk+1T∑i=1nτijk.Where uijk=νjk+p/νjk+δxi,μjk;Σjk and τijk=πjkfxi;μjk+1,νjk+1/fxi;Ψk+1.It can be seen thatμ^ik+1 and Σ^jk+1 were selected after weighted least squares estimation. The E-step drew the weights uijk while the M-step successfully chose μ^ik+1 and Σ^jk+1 using the weighted least squares method, which we called the EM algorithm.The t-distribution Algorithm.(1)
Input independent variablesx=x1,x2,...,xp of n sample.(2)
Assumex=x1,x2,...,xp to be the g-component normal mixture probability density function.(3)
Obtain the parameter mean and covariance to t-distribution.(4)
Implement EM algorithm at the iterationsk+1 th, πk+1,θk+1 , and νk+1.(5)
Compute the weight sample mean and sample covariance matrix asμ^jk+1 and Σ^jk+1.


## 4. Simulation and Results

The objective of this study was to classify the binary dependent variables using LDA and QDA via MM, ML, MVE, and t-distribution methods. The independent variables (x) were simulated from the multivariate normal distribution in two, four, six, and eight sets of independent variables and constant correlation (ρ) values of 0.1, 0.5, and 0.9. The multivariate normal distribution function of the independent variables (x) consisted of mean (μ) and covariance matrix (Σ):(26)fx|μ,Σ=12πp∑exp−12x−μT∑−1x−μ,where(27)x=xi1xi2...xip,μ=μ1μ2...μp,Σ=σ12ρσ1σ2...ρσ1σpρσ2σ1σ22...ρσ2σp............ρσpσ1ρσpσ2...σp2,p=2,4,6,8,i=1,...,n.The mean (μ) was fixed to generate at zero and the standard deviation (σi,i=1,2,...,p) was fixed to generate as 6. The regression coefficients were denoted by β=β0,β1,...,βpT for the set of two, four, six, and eight independent variables. Finally, the dependent variables (y) were approximated by the logit function πi=1/1+e−xβ from the logistic regression model.Ifπi≥0.5, the dependent variables were defined as yi=1, and yi=0, when πi<0.5. The R program employed simulation data at 1,000 replications with sample sizes of 200, 300, 400, and 500. The MM, ML, MVE, and t-distribution methods approximated LDA and QDA parameters to predict the dependent variable. The confusion matrix was used to report the classification performance on a set of estimated data to compare with the real data using the percentage of accuracy (Table 1).Table 1
The confusion matrix of real data and estimated data.
Estimated dataReal datayi=1yi=0y^ii=1True positive (TP)False positive (FP)y^i=0False negative (FN)True negative (TN)The percentage of accuracy was computed as (Table1): Percentage of Accuracy=TP+TN/TP+TN+FP+FN×100.The average percentage of accuracy of the MM, ML, MVE, and t-distribution methods for two, four, six and, eight variables are presented in Tables2–5 according to their correlation coefficient (0.1, 0.5, or 0.9). The mean of over 1,000 replications was used to compute the average percentage of accuracy. The maximum average percentage of accuracy values are shown in bold letters.Table 2
The average percentage of accuracy of moments (MM), maximum likelihood (ML), minimum value ellipsoid (MVE), and t-distribution (t) methods under two independent variables.
Sample sizes (n)Correlation CoefficientLinear discriminant analysisQuadratic discriminant analysisMMMLMVEtMMMLMVEt2000.198.1598.1597.6197.9298.1898.1898.0098.120.598.1298.1297.7097.9098.2798.2798.0098.210.998.1598.1597.7297.9398.2298.2298.0598.213000.198.4798.4798.1098.3198.5598.5598.3698.480.598.4598.4598.0798.2998.5098.5098.3998.490.998.3798.3898.0598.2198.4598.4698.4398.434000.198.5898.5898.2898.4998.6698.6698.6498.670.598.6798.6798.2898.5198.6998.6998.6298.660.998.6098.6098.3098.4698.6698.6698.6598.655000.198.7598.7598.4098.6198.7898.7898.7498.740.598.7298.7298.3798.5898.7698.7698.7398.730.998.7398.7398.3998.6098.8198.8198.7398.71Table 3
The average percentage of accuracy of moments (MM), maximum likelihood (ML), minimum value ellipsoid (MVE), and t-distribution (t) methods under four independent variables.
Sample sizes (n)Correlation CoefficientLinear discriminant analysisQuadratic discriminant analysisMMMLMVEtMMMLMVEt2000.197.5997.5997.3297.2997.4997.4996.7597.270.597.6097.6097.2897.4097.4997.6097.2897.360.997.6597.6597.3397.3997.5097.5096.5197.323000.197.9597.9597.7597.7497.8397.8397.3897.690.597.9797.9797.6597.7397.8197.8197.4297.660.997.9497.9497.7197.7497.8197.8297.4197.614000.198.1098.1098.9197.9097.9597.9697.7797.810.598.1098.1098.9697.9397.9397.9497.7297.840.998.2198.2198.0098.0598.0598.0697.8397.915000.198.3298.3298.0998.1198.1698.1797.9998.050.598.3198.3198.0998.1398.1698.1697.9398.030.998.3298.3298.1698.1398.1998.1997.9698.05Table 4
The average percentage of accuracy of moments (MM), maximum likelihood (ML), minimum value ellipsoid (MVE), and t-distribution (t) methods under six independent variables.
Sample sizes (n)Correlation CoefficientLinear discriminant analysisQuadratic discriminant analysisMMMLMVEtMMMLMVEt2000.197.2797.2797.0197.0096.9896.9895.7696.740.597.2697.2697.0396.9597.0797.0896.0096.800.997.3097.3097.0497.0497.1797.1896.0696.843000.197.5997.5997.4797.4697.2897.2996.7197.100.597.6197.6197.3997.3997.3297.3296.6397.090.997.8897.8897.3797.3797.2397.2396.6097.034000.197.8897.8897.6597.6597.4697.4696.9997.280.597.8497.8497.6097.6097.4597.4697.0097.260.997.8497.8597.6697.6697.5197.5197.0497.305000.198.0298.0297.8397.8397.6797.6797.3097.490.598.0298.0297.8397.8397.6697.6797.3097.460.998.0398.0397.8297.8297.6997.6997.2897.50Table 5
The average percentage of accuracy of moments (MM), maximum likelihood (ML), minimum value ellipsoid (MVE), and t-distribution (t) methods under eight independent variables.
Sample sizes (n)Correlation CoefficientLinear discriminant analysisQuadratic discriminant analysisMMMLMVEtMMMLMVEt2000.197.1197.1196.7296.8296.8596.8595.1296.410.597.0497.0496.7296.7496.8196.8195.1496.450.997.1397.1496.7896.8796.9096.9095.1896.473000.197.3897.3897.1097.0796.9296.9295.9996.650.597.2897.2897.1397.0596.9196.9295.9196.600.997.3297.3297.1297.0696.8996.9095.8896.614000.197.6297.6297.1797.3497.1797.1796.4796.910.597.5497.5497.3897.3097.1097.1096.3796.830.997.5797.5797.4097.3597.1697.1696.5096.905000.197.7497.7497.5697.5097.2797.2796.7997.300.597.7497.7497.5597.4997.3197.3196.7897.070.997.7797.7797.5897.6097.3297.3296.7797.08Tables2–5 present the average accuracy of all methods for several correlation coefficients. Based on the results, it could be seen that the average percentage of accuracy for the MM and ML methods had both the highest and equal values across all tables. The MM and ML methods in QDA also showed the highest average percentage of accuracy in Table 2. Additionally, the MM and ML methods in LDA had the highest average percentage in Tables 3–5. Meanwhile, the t-distribution method had the second highest average percentage of accuracy and a minor difference from the MM and ML methods. When the independent variable increased, the average percentage of accuracy decreased. When the sample sizes were large, the average percentage of accuracy was good. However, when the correlation coefficients increased, the average percentage of accuracy was shown to be a slightly different value.


## 5. Application of a Real Dataset

We applied LDA and QDA to classify a healthy person and a diabetic patient from the Pima Aboriginals diabetes dataset from the Applied Physics Laboratory of John Hopkins University. This dataset was obtained from the University of California, Irvine website (https://archive.ics.uci.edu/ml/datasets/). The binary dependent variables were 0 or 1, with 0 indicating a healthy person and 1 indicating a diabetic patient. The independent variables were defined by the number of pregnancies (x1), years of age (x2), diabetes pedigree function (x3), triceps skin fold thickness (x4), 2-h serum insulin (x5), plasma glucose concentration (x6), diastolic blood pressure (x7), and body mass index (kg/m2) (x8). These data consisted of 768 records: 500 healthy person records and 268 diabetic patient records. Table 6 presents the descriptive statistics related to the diabetic disease dataset.Table 6
Descriptive statistics of the diabetic patient dataset.
VariablesMeanMedianStd. Dev.MinMaxx13.843.003.360.0017.00x233.2429.0011.7621.0081.00x30.470.370.330.072.42x420.5423.0015.950.0099.00x579.8030.5115.240.00846.00x6120.90117.0031.970.00199.00x769.1072.0019.350.00122.00x831.9932.007.880.0067.10Pearson’s correlation analysis was used to determine whether there was a relationship between the eight continuous independent variables. The formula for computing the correlation between two variables was:(28)r=∑i=1nx−ix¯yi−y¯∑i=1nx−ix¯2∑i=1ny−iy¯2.The correlation coefficients of the independent variables are presented in Table7 and Figure 1. The hypothesis testing for the significance of the relationship is defined as:Table 7
Pearson correlation matrix for the relationship between eight independent variables.
Variablesx1x2x3x4x5x6x7x8x11.000.54∗−0.03−0.08−0.070.13∗0.14-->0.02p-value<0.051.000.210.33<0.05<0.051.00x2—1.000.03−0.11∗−0.040.260.23∗0.03p-value1.000.021.00<0.05<0.051.00x3——1.0000.18∗0.18∗0.13∗0.040.14∗p-value<0.05<0.05<0.051.00<0.05x4———1.0000.43∗0.050.20∗0.39∗p-value<0.050.79<0.05<0.05x5————1.0000.33∗0.080.19∗p-value<0.050.14<0.05x6—————1.0000.15∗0.22∗p-value<0.05<0.05x7——————1.0000.28∗p-value<0.05x8———————1.000p-valueNote.∗, The significant correlation at 0.05 level.Figure 1
The graph of the correlation matrix between eight independent variables.H0:ρ=0,H1:ρ≠0, and the t-statistics were used for computing the significance of Pearson’s correlation by t=rn−2/1−r2 with a degree of freedom (df) n-2. Finally, when the p-value of the t-statistics was less than 0.05, that indicated a significant relationship between two variables shown in Table 7.Based on our findings, a significant positive relationship at a moderate level was found in most cases such as inx1, x6 and x3 to x8, and a significant relationship at a strong level was shown in some cases such as x1, x2 and x4, x5. Most of the independent variables had a significant relationship except for x2, x3, x2, x4, and x2, x5.The Pearson correlation matrix from Table7 is rewritten in Figure 1, and is easier to understand using different shades. The light shade signifies a moderate correlation, and the dark shade denotes a strong correlation. Most of the independent variables have the light shade, which means there were either correlations among the independent variables or multicollinearity problems. The MM, ML, MVE, and t-distribution methods via LDA and QDA were approximated to compute the accuracy percentages in Table 8. The sets of two, four, six, or eight independent variables were similar to those of the simulation data. The two, four, and six independent variables were selected when the correlations were at the significant 0.05 level.Table 8
The percentage of prediction accuracy on two, four, six, and eight independent variables.
Number of independent variablesIndependent variablesLinear discriminant analysisQuadratic discriminant analysisMMMLMVEtMMMLMVEt2x1, x266.5366.5365.1066.5365.8865.8865.7567.44x1, x766.5366.5364.9766.5365.8865.8865.7567.44x3, x466.5366.5365.7566.5365.8865.8865.7567.44x5, x866.5366.5365.7566.5365.8865.8865.4967.44x6, x766.5366.5365.4966.5365.8865.8865.6267.44x7, x866.5366.5365.1066.5365.8865.8865.7567.444x1, x2, x4, x867.9667.9666.0167.4467.3167.3165.7566.92x2, x5, x6, x867.9667.9667.0567.4467.3167.3166.4066.92x2, x3, x4, x867.9667.9666.1467.4467.3167.3166.5366.92x3, x5, x7, x867.9667.9665.5767.4467.3167.3166.0166.926x2, x3, x5, x6, x7, x876.5176.5174.4776.0476.4376.4370.4472.39x1, x2, x3, x4, x6, x776.5276.5175.0076.0476.4376.4372.0072.398x1, x2, x3, x4, x5, x6, x7, x878.3878.3877.7777.2176.4376.5674.2173.3From Table8, it is apparent that the MM and ML methods showed equal accuracy percentages across all cases and the highest accuracy percentages in most cases. However, the accuracy percentage of the t-distribution method was good in two independent variables using QDA. Therefore, the MM and ML methods outperformed with LDA for four, six, and eight independent variables, which was the same as the simulation results. When the number of independent variables increased for two or four, the accuracy percentage was slightly different. However, the large independent variables demonstrated excellent performances in the diabetes dataset.


## 6. Discussion

The simulated results are listed in Tables2–5. The number of independent variables and sample sizes influenced the average accuracy percentage. QDA outperformed on the small independent variables, but the large independent variables were more appropriate for LDA. However, the average accuracy percentages of the large independent variables were less than the percentages for the small independent variables. The correlation coefficient increase did not affect the classification because the average accuracy percentage showed slightly different values. If the sample size increased, the accuracy of all methods increased in all cases.For the real data results in Table8, the MM, ML, and t-distribution methods showed the highest accuracy percentages for two independent variables. We found that the independent variables of real data demonstrated skewed data (Figure 2). The Shapiro-Wilk test [22] was used to confirm that all independent variables showed non-normality. However, the MM and ML methods supported classification on large independent variables. The multivariate normal distribution generated the simulation dataset. Therefore, the real data results were different from the simulation study in some cases. Overall, it is clear that the MM and ML methods are most suitable for classifying diabetes.Figure 2
The histogram of eight independent variables.The Pima Indian diabetic dataset is popular to analyze for classification and to improve medical diagnosis. Gupta et al. [23] compared Gaussian process, LDA, QDA, statistical gradient descent, ridge regression classifier, support vector machines, k-nearest neighbors, decision tree, naïve Bayes, logistic regression, random forest, and ELM for multiquadric, RBF, sigmoid activation functions. The results suggested that the logistic regression performs better than the other techniques. Chang et al. [24] proposed an e-diagnosis system based on a machine learning algorithm: Naïve Bayes classifier, random forest classifier, and J48 decision tree models to be trained and tested using the Pima Indians diabetes. It can be concluded that a Naïve Bayes model works well with a more fine-tuned selection of features for binary classification. Abedin et al. [25] studied a hierarchical ensemble model to combine two classifiers that had been trained, a decision tree and a logistic regression model, and feed the output of those models to a neural network. The proposed model achieved classification accuracy by using PIMA Indian diabetes database.


## 7. Conclusion

This paper proposes the classification of binary data by applying MM, ML, MVE, and t-distribution methods based on LDA and QDA for data multicollinearity. The solutions provide the advantages and disadvantages of these methods. The advantage of the QDA results when using small independent variables, MM and ML, outperformed in classifying all correlation coefficients, but these methods are slightly different from the other methods. However, the LDA results showed that MM and ML perform the same when provided with four to eight independent variables. These methods did not relate to the classification performance when considering the correlation coefficient. However, the large sample sizes showed good performances across all cases. Eight independent variables were selected for the medical data when using real data to classify diabetes patients. We selected two, four, six, and eight independent variables for large correlation. These results showed that the t-distribution was efficient on two independent variables via QDA. However, the MM and ML methods were effective at classifying four to eight variables based on LDA. Therefore, we concluded that MM and ML methods could classify in the presence of multicollinearity. Medical data classification has been extended to study a novel random vector functional link [26] and a novel random vector functional link with ɛ-insensitive Huber loss function [27]. A fuzzy-based Lagrangian twin parametric-margin support vector machine [28] reduced the effect of the outliers in medical data.



---

*Source: 7829795-2022-09-06.xml*