# Credit Risk Evaluation with a Least Squares Fuzzy Support Vector Machines Classifier

Authors: Lean Yu
Journal: Discrete Dynamics in Nature and Society
 (2014)

---

## Abstract

A least squares fuzzy support vector machine (LS-FSVM) model that integrates advantages of fuzzy support vector machine (FSVM) and least squares method is proposed for credit risk evaluation. In the proposed LS-FSVM model, the purpose of incorporating the concepts of fuzzy sets is to
add generalization capability and outlier insensitivity, while the least squares method is adopted to reduce the computational complexity. For illustrative
purposes, a real-world credit risk dataset is used to test the effectiveness and robustness of the proposed LS-FSVM methodology.


---

## Body

## 1. Introduction

Credit risk evaluation has been a major area of focus for financial and banking industries due to recent financial crises as well as the Basel III regulations. Since the seminal work of Altman [1] was published, many different techniques such as discriminant analysis [1], logit analysis [2], probit analysis [3], linear programming [4], integer programming [5], k-nearest neighbor (KNN) classifiers [6], and classification trees [7] have widely been applied to credit risk assessment tasks. With the advance of modern computing technology, some artificial intelligence (AI) tools such as artificial neural networks (ANN) [8, 9], genetic algorithms (GAs) [10, 11], self-organizing learning [12], support vector machines (SVMs) [13–18], and some variants of SVMs [19–22] have also been employed for credit risk evaluation. Empirical results have revealed that these AI techniques offer advantages over traditional statistical models and optimization techniques in credit risk evaluation.According to the literature review, it is easy to find that almost all classification methods can be used for credit risk assessment. However, some hybrid and combined (or ensemble) classifiers, which integrate two or more single classification methods, have shown higher predictability than individual methods. Research on combined or ensemble classifiers is currently flourishing in credit risk evaluation. Recent examples are neural discriminant technique [23], neurofuzzy [16, 24], fuzzy SVM [25], rough set based SVM [26], evolving neural network [27], neural network ensemble [28, 29], support vector machine based multiagent ensemble learning [30], and AI-based fuzzy group decision making (GDM) model [31]. Two recent surveys [32, 33] and one monograph [34] cover credit risk analysis in more detail.In this study, a new credit classification technique, least squares fuzzy SVM (LS-FSVM), is proposed to discriminate good from bad customers in customer credit evaluation. In the existing studies, the fuzzy SVM (FSVM) proposed by Lin and Wang [35] has been shown to be suitable for customer credit assessment [34]. The main reason is that in credit risk evaluation we usually cannot label a customer as absolutely good one who is sure to repay in time or absolutely bad one who will default certainly, while the FSVM treats every sample as belonging to good and bad classes to some extent. This enables the FSVM to offer a higher generalization capability without losing the merits of insensitivity to outliers. Although the FSVM has good generalization capabilities and outlier insensitivity, computational complexity of the FSVM makes its use rather difficult because the final solution of FSVM is derived from a quadratic programming (QP) problem [35]. To reduce the computational complexity of FSVM, this study applies the least squares method to reduce the computational complexity of FSVM and to formulate a new classification method—least squares FSVM (LS-FSVM). In the proposed LS-FSVM model, equality constraints instead of inequality constraints are used. As a result, the solution is obtained from a set of linear equations, instead of QP presenting in the classical FSVM approach [35], thereby reducing the computational complexity, relative to the FSVM.From the above descriptions, the main advantage of the proposed LS-FSVM can be summarized with regard to the following two aspects. On the one hand, fuzzification processing can increase the generalization capability and improve the suitability of SVM due to the fact that the same uncertainties can be well treated by fuzzy membership. On the other hand, the least squares method can reduce the computational complexity of FSVM because the solution of least squares FSVM can be obtained from a set of linear equations instead of a QP problem, thus increasing the computational speed of FSVM which is attractive in solving fuzzy information engineering problems. In the existing literature, Tsujinishi and Abe [36] proposed a fuzzy LS-SVMs method to solve the multiclass problems. Regrettably, the performance of the proposed fuzzy LS-SVMs model is inferior to the fuzzy SVMs. On the contrary, the proposed LS-FSVM model in this paper reported the good performance in the two-class problems due to the above features of LS-FSVM model.The main motivation of this study is to formulate the least squares version of FSVM for binary classification problems and to compare its performance with some typical classification techniques in the area of credit risk evaluation. The rest of this study is organized as follows. Section2 illustrates the formulation of the LS-FSVM methodology. In Section 3, a real-world credit dataset is used to test the performance of the LS-FSVM to classify different samples. Section 4 concludes the paper.


## 2. Methodology Formulation

In this section, a brief introduction of SVM classifiers [37] is first presented. Then a fuzzy SVM (FSVM) model [35] is briefly reviewed. Finally, the least squares FSVM (LS-FSVM) model is formulated.


### 2.1. SVM for Binary Classification (By Vapnik [37])

Given a training dataset{xi,yi}(i=1,…,N), where xi∈RN is the ith input pattern and yi is its corresponding observed result, which is a binary variable. In credit risk evaluation models, xis denote the attributes of customers and yi is the observed outcome of repayment obligations. If the customer defaults, yi=1, or else yi=-1. The generic idea of SVM is first to map the input data into a high-dimensional feature space through a mapping function and then to find the optimal separating hyperplane with minimal classification errors. The separating hyperplane can be represented as
(1)z(x)=wTϕ(x)+b=0,
where w is the normal vector of the hyperplane and b is the bias.Supposeϕ(·) is a nonlinear function that maps the input space into a higher dimensional feature space. If a dataset is linearly separable in this feature space, the classifier is constructed as
(2)wTϕ(xi)+b≥1ifyi=1wTϕ(xi)+b≤-1ifyi=-1
which is equivalent to
(3)yi(wTϕ(xi)+b)≥1fori=1,…,N.In order to deal with dataset that is not linearly separable, the previous analysis can be generalized by introducing some nonnegative variablesξi≥0, such that (3) can be modified as follows:
(4)yi[wTϕ(xi)+b]≥1-ξifori=1,…,N.The nonnegativeξi in (4) are those for which data point xi does not satisfy (3). Thus, the term ∑i=1N‍ξi can be considered as a measure of the amount of misclassification, that is, tolerable misclassification errors.According to the structural risk minimization principle [37], the risk bound is minimized by solving the following optimization problem:
(5)MinimizeΦ(w,b,ξi)=12wTw+C∑i=1N‍ξiSubjectto:yi(wTϕ(xi)+b)=1-ξifori=1,…,Nhhhhhhhhhξi≥0fori=1,…,N,
where C is a free regularization parameter controlling a tradeoff between margin maximization and tolerable misclassification errors.Searching the optimal hyperplane in (5) is a quadratic programming (QP) problem. After introducing a set of Lagrangian multipliers αi and βi for constraints in (5), the primal problem in (5) is to find out the saddle point of the Lagrangian function; that is,
(6)min⁡w,b,ξiL(w,b,ξi;αi,βi)=12wTw+C∑i=1N‍ξi-∑i=1N‍αi[yi(wTϕ(xi)+b)-1+ξi]-∑i=1N‍βiξi.Differentiating (6) with w, b, and ξi, one obtains
(7)ddwL(w,b,ξi;αi,βi)=w-∑i=1N‍αiyiϕ(xi)=0,ddbL(w,b,ξi;αi,βi)=-∑i=1N‍αiyi=0,ddξiL(w,b,ξi;αi,βi)=C-αi-βi=0.From (7) one has w=∑i=1Nαiyiϕ(xi). The key issue is how to determine the value of αi. To obtain a solution, the dual problem of the primal problem (6) becomes
(8)MaximizeQ(α)=-12∑i=1N‍∑j=1N‍αiαjyiyjϕ(xi)Tϕ(xj)hhhhhhhhhhhhhhh+∑i=1N‍αiSubjectto:∑i=1N‍αiyi=0,0≤αi≤C,i=1,…,N.Functionϕ(x) in (8) is then related to K(xi,xj) by imposing
(9)ϕ(xi)Tϕ(xi)=K(xi,xj),
which is motivated by the Mercer theorem [37]. K(xi,xj) is the kernel function in the input space that determines the inner products of the two data points in the feature space. According to the Karush-Kuhn-Tucker (KKT) theorem [38], the KKT conditions are defined as
(10)αi[yi(wTϕ(x)+b)-1+ξi]=0,i=1,2,…,N,(11)(C-αi)ξi=0,i=1,2,…,N.From this equality, it is deduced that the only nonzero values,αi, in (10), are those for which constraints in (4) are satisfied with the equality sign. Data points xi(i=1,2,…,s), corresponding to αi>0, are called support vectors (SVs). But there are two types of SVs in a nonseparable case. In the case of 0<αi<C, the corresponding support vector xi satisfies equalities yk[wTϕ(xk)+b]=1 and ξi=0. In the case of αi=C, the corresponding ξi is not zero and the corresponding support vector xi does not satisfy (2), which considered such SVs as misclassification errors. In terms of the above processing, data points xi(i=1,2,…,s), corresponding to αi=0, are classified correctly.Using the support vectors, the optimal solution for the weight vector in (7) can be given by
(12)w=∑i=1Ns‍αiyiϕ(xi),
where Ns is the number of SVs. Moreover, in the case of 0<αi<C, condition ξi=0 applies to (11) in terms of the KKT theorem [38]. Thus, one may determine the optimal bias b by taking any data point in the dataset. However, from the numerical perspective, it is better to take the mean value of b, from such data points in the dataset. Once the optimal parameter p air (w,b) is determined, the decision function of the SVM classifiers can be represented as
(13)z(x)=sign⁡(∑i=1Ns‍αiyiK(xi,xj)+b).


### 2.2. FSVM (by Lin and Wang [35])

SVM has been proved to be a powerful tool for solving classification problems [37], but it has some inherent limitations. From its formulation discussed above, each training point in the training dataset belongs to either one or the other class. But in many real-world applications, every training sample does not exactly belong to one of the two classes; it may belong to one class to the extent of 80 percent and 20 percent of it may be meaningless. That is to say, there is a membership grade {μi}i=1N∈[0,1] associated with each training data point xi. In this sense, FSVM is an extension of a SVM that takes into consideration the varying significance of the training samples. For FSVM, each training sample is associated with a membership value {μi}i=1N∈[0,1]. The membership value μi reflects the confidence degree of the data points. The higher the value, the higher the confidence degree of its class label. Similar to SVM, the optimization problem of the FSVM [35] is formulated as follows:
(14)MinimizeΨ(w,b,ξi,μi)=12wTw+C∑i=1N‍μiξiSubjectto:yi(wTϕ(xi)+b)≥1-ξifori=1,…,Nhhhhhhhhhhξi≥0fori=1,…,N.Similar to SVM, the solution of FSVM is obtained from the above quadratic programming (QP) problem. Note that error termξi is scaled by membership value μi. The membership values used to weigh the soft penalty term reflect the relative confidence degree of the training samples during training. Important samples with larger membership values will have more impact in FSVM training than those with smaller values.Similar to Vapnik’s SVM [37], optimization problem of the FSVM can be transformed into the following dual problem:
(15)MaximizeW(α)=-12∑i=1N‍∑j=1N‍αiαjyiyjK(xi,xj)hihhhhhhhhhhhhhh+∑i=1N‍αiSubjectto:∑i=1N‍αiyi=0,0≤αi≤μiC,i=1,…,N.In the same way, KKT conditions are defined as(16)αi[yi(wTϕ(x)+b)-1+ξi]=0,i=1,2,…,N,(μiC-αi)ξi=0,i=1,2,…,N.Data pointxi, corresponding to αi>0, is called a support vector. There are two types of SVs. The one corresponding to 0<αi<μiC lies on the margin of the hyperplane, and the other corresponding to αi=μiC is treated as misclassified.Solving (15) leads to a decision function similar to (13), but with different support vectors and the corresponding weights αi. An important difference between SVM and FSVM is that data points with the same value of αi may be indicated as a different type of SVs in FSVM due to membership factor μi. Interested readers can refer to [35] for more details.


### 2.3.  Least Squares FSVM

In both SVM and FSVM, the final solution can be described as an issue closely associated with quadratic programming (QP) problem. The main issue with the QP method is that it is a time-consuming process to find the solutions when handling some large-scale real-world application problems. Motivated by Lai et al. [14] and Suykens and Vandewalle [39], the least squares FSVM (LS-FSVM) model is introduced by formulating the following optimization problem:
(17)Minimizeφ(w,b,ξi,μi)=12wTw+C2∑i=1N‍μiξi2Subjectto:yi(wTϕ(xi)+b)=1-ξifori=1,…,N.One can define the Lagrangian function as(18)min⁡w,b,ξiL(w,b,ξi;αi)=12wTw+C2∑i=1N‍μiξi2-∑i=1N‍αi[yi(wTϕ(xi)+b)-1+ξi],
where αi is the ith Lagrangian multiplier, which can be either positive or negative, due to equality constraints in accordance with KKT conditions [38].The optimal conditions are obtained by differentiating (18):
(19)ddwL(w,b,ξi;αi)=w-∑i=1N‍αiyiϕ(xi)=0,ddbL(w,b,ξi;αi)=-∑i=1N‍αiyi=0,ddξiL(w,b,ξi;αi)=μiCξi-αi=0,ddαiL(w,b,ξi;αi)=yi[wTϕ(xi)+b]-1+ξi=0.From (19), one can obtain the following equations:
(20)w=∑i=1N‍αiyiϕ(xi),∑i=1N‍αiyi=0,αi=μiCξi,yi[wTϕ(xi)+b]-1+ξi=0.Using a matrix form, optimal conditions in (20) can be expressed by
(21)[ΩYYT0][αb]=[10],
where Ω,Y, and1 are (22), (23), and (24), respectively:
(22)Ωij=yiyjϕ(xi)Tϕ(xj)+(μiC)-1I(23)Y=(y1,y2,…,yN)T,(24)1=(1,1,…,1)T.From (22), Ω is positive definite. Thus, α can be obtained from (21); that is,
(25)α=Ω-1(1-bY).Substituting (25) into the second matrix equation in (21), we can obtain
(26)b=YTΩ-11YTΩ-1Y.Here, sinceΩ is positive definite, Ω-1 is also positive definite. In addition, since Y is a nonzero vector, YTΩ-1Y>0. Thus, b is always obtained. Substituting (26) into (25), α can be obtained.Hence, the separating hyperplane of LS-FSVM can be found by solving the linear set of (21)–(24), instead of quadratic programming (QP), thereby reducing the computational complexity, especially for large-scale problems.The main advantages of the LS-FSVM can be summarized into the following five aspects. First of all, the LS-FSVM requires fewer prior assumptions about the input data than those required in statistical approaches, such as normal distribution and continuity. Second, it can perform nonlinear mapping from an original input space into a high dimensional feature space, in which it constructs a linear discriminant function to replace the nonlinear function in the original input space. This characteristic also solves the dimension disaster problem because its computational complexity is not dependent on the samples’ dimension. Third, it attempts to learn the separating hyperplane to maximize the classification margin, thereby implementing structural risk minimization and realizing good generalization capability. Fourth, a distinct trait of LS-FSVM is that it can further lower computational complexity by transforming a quadratic programming problem into a linear equation set problem. Finally, an important advantage of LS-FSVM is that support valuesαi in LS-FSVM are proportional to the membership degree, as well as misclassification errors at the data points, thus making LS-FSVM more suitable for some real-world problems, which is the main difference between the proposed LS-FSVM and the traditional SVM and LS-SVM models. These important characteristics also make LS-FSVM preferable in many practical applications. In the following section, some experiments are presented for verification purpose.


## 2.1. SVM for Binary Classification (By Vapnik [37])

Given a training dataset{xi,yi}(i=1,…,N), where xi∈RN is the ith input pattern and yi is its corresponding observed result, which is a binary variable. In credit risk evaluation models, xis denote the attributes of customers and yi is the observed outcome of repayment obligations. If the customer defaults, yi=1, or else yi=-1. The generic idea of SVM is first to map the input data into a high-dimensional feature space through a mapping function and then to find the optimal separating hyperplane with minimal classification errors. The separating hyperplane can be represented as
(1)z(x)=wTϕ(x)+b=0,
where w is the normal vector of the hyperplane and b is the bias.Supposeϕ(·) is a nonlinear function that maps the input space into a higher dimensional feature space. If a dataset is linearly separable in this feature space, the classifier is constructed as
(2)wTϕ(xi)+b≥1ifyi=1wTϕ(xi)+b≤-1ifyi=-1
which is equivalent to
(3)yi(wTϕ(xi)+b)≥1fori=1,…,N.In order to deal with dataset that is not linearly separable, the previous analysis can be generalized by introducing some nonnegative variablesξi≥0, such that (3) can be modified as follows:
(4)yi[wTϕ(xi)+b]≥1-ξifori=1,…,N.The nonnegativeξi in (4) are those for which data point xi does not satisfy (3). Thus, the term ∑i=1N‍ξi can be considered as a measure of the amount of misclassification, that is, tolerable misclassification errors.According to the structural risk minimization principle [37], the risk bound is minimized by solving the following optimization problem:
(5)MinimizeΦ(w,b,ξi)=12wTw+C∑i=1N‍ξiSubjectto:yi(wTϕ(xi)+b)=1-ξifori=1,…,Nhhhhhhhhhξi≥0fori=1,…,N,
where C is a free regularization parameter controlling a tradeoff between margin maximization and tolerable misclassification errors.Searching the optimal hyperplane in (5) is a quadratic programming (QP) problem. After introducing a set of Lagrangian multipliers αi and βi for constraints in (5), the primal problem in (5) is to find out the saddle point of the Lagrangian function; that is,
(6)min⁡w,b,ξiL(w,b,ξi;αi,βi)=12wTw+C∑i=1N‍ξi-∑i=1N‍αi[yi(wTϕ(xi)+b)-1+ξi]-∑i=1N‍βiξi.Differentiating (6) with w, b, and ξi, one obtains
(7)ddwL(w,b,ξi;αi,βi)=w-∑i=1N‍αiyiϕ(xi)=0,ddbL(w,b,ξi;αi,βi)=-∑i=1N‍αiyi=0,ddξiL(w,b,ξi;αi,βi)=C-αi-βi=0.From (7) one has w=∑i=1Nαiyiϕ(xi). The key issue is how to determine the value of αi. To obtain a solution, the dual problem of the primal problem (6) becomes
(8)MaximizeQ(α)=-12∑i=1N‍∑j=1N‍αiαjyiyjϕ(xi)Tϕ(xj)hhhhhhhhhhhhhhh+∑i=1N‍αiSubjectto:∑i=1N‍αiyi=0,0≤αi≤C,i=1,…,N.Functionϕ(x) in (8) is then related to K(xi,xj) by imposing
(9)ϕ(xi)Tϕ(xi)=K(xi,xj),
which is motivated by the Mercer theorem [37]. K(xi,xj) is the kernel function in the input space that determines the inner products of the two data points in the feature space. According to the Karush-Kuhn-Tucker (KKT) theorem [38], the KKT conditions are defined as
(10)αi[yi(wTϕ(x)+b)-1+ξi]=0,i=1,2,…,N,(11)(C-αi)ξi=0,i=1,2,…,N.From this equality, it is deduced that the only nonzero values,αi, in (10), are those for which constraints in (4) are satisfied with the equality sign. Data points xi(i=1,2,…,s), corresponding to αi>0, are called support vectors (SVs). But there are two types of SVs in a nonseparable case. In the case of 0<αi<C, the corresponding support vector xi satisfies equalities yk[wTϕ(xk)+b]=1 and ξi=0. In the case of αi=C, the corresponding ξi is not zero and the corresponding support vector xi does not satisfy (2), which considered such SVs as misclassification errors. In terms of the above processing, data points xi(i=1,2,…,s), corresponding to αi=0, are classified correctly.Using the support vectors, the optimal solution for the weight vector in (7) can be given by
(12)w=∑i=1Ns‍αiyiϕ(xi),
where Ns is the number of SVs. Moreover, in the case of 0<αi<C, condition ξi=0 applies to (11) in terms of the KKT theorem [38]. Thus, one may determine the optimal bias b by taking any data point in the dataset. However, from the numerical perspective, it is better to take the mean value of b, from such data points in the dataset. Once the optimal parameter p air (w,b) is determined, the decision function of the SVM classifiers can be represented as
(13)z(x)=sign⁡(∑i=1Ns‍αiyiK(xi,xj)+b).


## 2.2. FSVM (by Lin and Wang [35])

SVM has been proved to be a powerful tool for solving classification problems [37], but it has some inherent limitations. From its formulation discussed above, each training point in the training dataset belongs to either one or the other class. But in many real-world applications, every training sample does not exactly belong to one of the two classes; it may belong to one class to the extent of 80 percent and 20 percent of it may be meaningless. That is to say, there is a membership grade {μi}i=1N∈[0,1] associated with each training data point xi. In this sense, FSVM is an extension of a SVM that takes into consideration the varying significance of the training samples. For FSVM, each training sample is associated with a membership value {μi}i=1N∈[0,1]. The membership value μi reflects the confidence degree of the data points. The higher the value, the higher the confidence degree of its class label. Similar to SVM, the optimization problem of the FSVM [35] is formulated as follows:
(14)MinimizeΨ(w,b,ξi,μi)=12wTw+C∑i=1N‍μiξiSubjectto:yi(wTϕ(xi)+b)≥1-ξifori=1,…,Nhhhhhhhhhhξi≥0fori=1,…,N.Similar to SVM, the solution of FSVM is obtained from the above quadratic programming (QP) problem. Note that error termξi is scaled by membership value μi. The membership values used to weigh the soft penalty term reflect the relative confidence degree of the training samples during training. Important samples with larger membership values will have more impact in FSVM training than those with smaller values.Similar to Vapnik’s SVM [37], optimization problem of the FSVM can be transformed into the following dual problem:
(15)MaximizeW(α)=-12∑i=1N‍∑j=1N‍αiαjyiyjK(xi,xj)hihhhhhhhhhhhhhh+∑i=1N‍αiSubjectto:∑i=1N‍αiyi=0,0≤αi≤μiC,i=1,…,N.In the same way, KKT conditions are defined as(16)αi[yi(wTϕ(x)+b)-1+ξi]=0,i=1,2,…,N,(μiC-αi)ξi=0,i=1,2,…,N.Data pointxi, corresponding to αi>0, is called a support vector. There are two types of SVs. The one corresponding to 0<αi<μiC lies on the margin of the hyperplane, and the other corresponding to αi=μiC is treated as misclassified.Solving (15) leads to a decision function similar to (13), but with different support vectors and the corresponding weights αi. An important difference between SVM and FSVM is that data points with the same value of αi may be indicated as a different type of SVs in FSVM due to membership factor μi. Interested readers can refer to [35] for more details.


## 2.3.  Least Squares FSVM

In both SVM and FSVM, the final solution can be described as an issue closely associated with quadratic programming (QP) problem. The main issue with the QP method is that it is a time-consuming process to find the solutions when handling some large-scale real-world application problems. Motivated by Lai et al. [14] and Suykens and Vandewalle [39], the least squares FSVM (LS-FSVM) model is introduced by formulating the following optimization problem:
(17)Minimizeφ(w,b,ξi,μi)=12wTw+C2∑i=1N‍μiξi2Subjectto:yi(wTϕ(xi)+b)=1-ξifori=1,…,N.One can define the Lagrangian function as(18)min⁡w,b,ξiL(w,b,ξi;αi)=12wTw+C2∑i=1N‍μiξi2-∑i=1N‍αi[yi(wTϕ(xi)+b)-1+ξi],
where αi is the ith Lagrangian multiplier, which can be either positive or negative, due to equality constraints in accordance with KKT conditions [38].The optimal conditions are obtained by differentiating (18):
(19)ddwL(w,b,ξi;αi)=w-∑i=1N‍αiyiϕ(xi)=0,ddbL(w,b,ξi;αi)=-∑i=1N‍αiyi=0,ddξiL(w,b,ξi;αi)=μiCξi-αi=0,ddαiL(w,b,ξi;αi)=yi[wTϕ(xi)+b]-1+ξi=0.From (19), one can obtain the following equations:
(20)w=∑i=1N‍αiyiϕ(xi),∑i=1N‍αiyi=0,αi=μiCξi,yi[wTϕ(xi)+b]-1+ξi=0.Using a matrix form, optimal conditions in (20) can be expressed by
(21)[ΩYYT0][αb]=[10],
where Ω,Y, and1 are (22), (23), and (24), respectively:
(22)Ωij=yiyjϕ(xi)Tϕ(xj)+(μiC)-1I(23)Y=(y1,y2,…,yN)T,(24)1=(1,1,…,1)T.From (22), Ω is positive definite. Thus, α can be obtained from (21); that is,
(25)α=Ω-1(1-bY).Substituting (25) into the second matrix equation in (21), we can obtain
(26)b=YTΩ-11YTΩ-1Y.Here, sinceΩ is positive definite, Ω-1 is also positive definite. In addition, since Y is a nonzero vector, YTΩ-1Y>0. Thus, b is always obtained. Substituting (26) into (25), α can be obtained.Hence, the separating hyperplane of LS-FSVM can be found by solving the linear set of (21)–(24), instead of quadratic programming (QP), thereby reducing the computational complexity, especially for large-scale problems.The main advantages of the LS-FSVM can be summarized into the following five aspects. First of all, the LS-FSVM requires fewer prior assumptions about the input data than those required in statistical approaches, such as normal distribution and continuity. Second, it can perform nonlinear mapping from an original input space into a high dimensional feature space, in which it constructs a linear discriminant function to replace the nonlinear function in the original input space. This characteristic also solves the dimension disaster problem because its computational complexity is not dependent on the samples’ dimension. Third, it attempts to learn the separating hyperplane to maximize the classification margin, thereby implementing structural risk minimization and realizing good generalization capability. Fourth, a distinct trait of LS-FSVM is that it can further lower computational complexity by transforming a quadratic programming problem into a linear equation set problem. Finally, an important advantage of LS-FSVM is that support valuesαi in LS-FSVM are proportional to the membership degree, as well as misclassification errors at the data points, thus making LS-FSVM more suitable for some real-world problems, which is the main difference between the proposed LS-FSVM and the traditional SVM and LS-SVM models. These important characteristics also make LS-FSVM preferable in many practical applications. In the following section, some experiments are presented for verification purpose.


## 3.  Experiment Analysis

In this section, a real-world credit dataset is used to test the performance of LS-FSVM. For comparison purposes, linear regression (LinR) [14], logistic regression (LogR) [2], artificial neural network (ANN) [8, 9], Vapnik’s SVM [37], Lin and Wang’s FSVM [35], and LS-SVM [39] are also used.The dataset in this study comes from a financial services company of England, obtained from a CDROM published by Thomas et al. [40]. Each applicant has 14 characteristics or variables, listed in Table 1. The dataset includes detailed information of 1225 applicants, of which 323 are observed as bad customers.Table 1
Variables of the dataset.
Number
Variables
1
Year of birth
2
Number of children
3
Number of other dependents
4
Is there a home phone
5
Applicant’s income
6
Applicant’s employment status
7
Spouse’s income
8
Residential status
9
Value of home
10
Mortgage balance outstanding
11
Outgoings on mortgage or rent
12
Outgoings on loans
13
Outgoings on hire purchase
14
Outgoings on credit cardsIn this experiment, LS-FSVM, FSVM, LS-SVM, and SVM models use RBF kernel for classification. In the ANN model, a three-layer back-propagation neural network with 10 sigmoidal neurons is used in the hidden layer, and one linear neuron is used in the output layer. The network training algorithm is the Levenberg-Marquardt (LM) algorithm. Besides, the learning and momentum rates are set to 0.1 and 0.15, respectively. The accepted average squared error is 0.05, and the training epochs are 1600. The above parameters are obtained by trial-and-error method. The experiment is run by MATLAB 6.1 with statistical toolbox, NNET toolbox, and LS-SVM toolbox. In addition, three evaluation criteria are used to measure the efficiency of classification:(27)TypeIaccuracy=numberofbothobservedbadandclassifiedasbadnumberofobservedbadTypeIIaccuracy=numberofbothobservedgoodandclassifiedasgoodnumberofobservedgoodTotalaccuracy=numberofcorrectclassificationthenumberofevaluationsample.To show the classification capability of LS-FSVM in distinguishing potentially insolvent customers from good customers, we perform the test with LS-FSVM at the beginning. This testing process includes five steps.First, the number of observed bad customers is tripled to make their number nearly equal to the number of observed good customers. The main purpose of such processing is to avoid the impact on performance of imbalanced samples. A similar processing method can be found in Wang et al. [25]. Of course, some other imbalanced data processing methods, for example, the information granulation based method [41], can also be used.Second, the original data is preprocessed to impute the missing data and transform category data. In this study, interpolation method is used to impute the missing data. For category data, a numerical method is used for transformation.Third, the original dataset is randomly separated into two parts, that is, training samples and testing samples. In this study, 1500 samples are used for training and the remaining 371 samples are used for holdout testing and performance evaluation.Fourth, membership grades are generated by the linear transformation function proposed by Wang et al. [25], in terms of the initial score obtained by experts’ experience and opinions. Of course, some new membership generation methods in [42] can also be adopted.Finally, the LS-FSVM classifier is trained and accordingly the results can be evaluated. The above five steps are repeated 20 times to confirm the robustness of the proposed method. In this study, the efficiency and robustness of credit risk evaluation using the LS-FSVM model are shown in Table2.Table 2
Computational results of credit risk evaluation using LS-FSVM.
Experiment number
Type I (%)
Type II (%)
Total (%)
1
81.56
93.81
88.54
2
86.98
95.14
92.53
3
82.02
91.84
88.49
4
79.81
98.36
93.53
5
87.77
94.03
92.24
6
81.56
96.85
92.38
7
79.19
93.05
87.41
8
85.69
92.11
88.86
9
79.27
89.33
87.63
10
82.45
96.58
90.45
11
77.96
93.08
85.06
12
80.61
89.57
85.89
13
81.16
93.41
88.34
14
78.96
89.88
87.56
15
85.56
97.35
94.53
16
70.36
98.13
84.11
17
80.14
92.01
88.68
18
75.22
89.54
86.25
19
86.72
89.61
88.04
20
83.85
94.70
92.64
Mean
81.34
93.41
89.21
Stdev
4.20
2.98
3.02As can be seen from Table2, the proposed LS-FSVM model exhibits significant classification capabilities. In the 20 experiments, type I accuracy, type II accuracy, and total accuracy are 81.34%, 93.41%, and 89.21%, respectively, in the mean sense. Furthermore, the standard deviation is rather small, revealing that robustness of the LS-FSVM classifier is good. These results imply that the LS-FSVM model is a promising credit risk evaluation technique.For further illustration, the classification power of the LS-FSVM is also compared with six other commonly used classifiers: linear regression (LinR) [14], logistics regression (LogR) [2], artificial neural network (ANN) [8, 9], Vapnik’s SVM [37], FSVM [35], and LS-SVM [39]. The results of the comparison are reported in Table 3.Table 3
Results of comparisons of different classifiers.
Method
Type I (%)
Type II (%)
Overall (%)
CPU time (s)
LinR
52.87
43.48
50.22
102.45
LogR
60.08
62.29
60.66
112.87
ANN
56.57
78.36
72.24
318.72
SVM
70.13
83.49
77.02
306.33
LS-SVM
79.37
93.27
89.16
138.61
FSVM
80.08
92.86
88.38
335.24
LS-FSVM
81.34
93.41
89.21
149.49From Table3, several important results can be observed.(a)
For type I accuracy, LS-FSVM is the best of all the listed approaches, followed by FSVM, LS-SVM, Vapnik’s SVM, logistics regression, artificial neural network model, and linear regression model, implying that the LS-FSVM is a very promising technique in credit risk assessment. Particularly, performance of the two fuzzy SVM techniques (Lin and Wang’s FSVM [35] and LS-FSVM) is better than other classifiers listed in this study, implying that the fuzzy SVM classifier may be more suitable for credit risk assessment tasks than other deterministic classifiers, such as linear regression (LinR) and logit regression (LogR).(b)
From the viewpoint of type II accuracy, LS-FSVM and LS-SVM outperform the other five models, implying the strong capability of the least squares version of the SVM model in credit risk evaluation. In the meantime, the proposed LS-FSVM model seems to be slightly better than LS-SVM, revealing that the LS-FSVM is a feasible solution to improve the accuracy of credit risk evaluation. Interestingly, performance of the FSVM is slightly worse than that of the LS-SVM. The reasons for this phenomenon are worth exploring further.(c)
According to the total accuracy, performances of the two statistical models (LinR and LogR) are much worse than those of the other five models. The main reason is that the five models can effectively capture the nonlinear patterns hidden in the credit data. As is known, there are many factors that affect customer credit. Usually, the relationships between customer default and these factors are very subtle and complex. Besides some linear relationships, some nonlinear relationships often exist in the credit data. Therefore, nonlinear intelligent models can offer advantage over traditional linear models.(d)
From the perspective of computational time, two traditional classification models (i.e., LinR and LogR) are faster than all the intelligent models due to their simplicity. In all the intelligent models, LS-SVM and LS-FSVM are faster than other intelligent models due to the least squares principle. However, LS-FSVM is slightly slower than LS-SVM because the fuzzification needs some processing time, but LS-FSVM performs faster than the SVM and FSVM, indicating that the proposed LS-FSVM model is a very effective model in credit risk evaluation.(e)
In the five intelligent models, the performance of ANN and SVM models is much worse than the other three intelligent models. The main reason is that the standard ANN and SVM models have their own shortcomings, such as the sensitivity of parameters and outliers, thus affecting their classification performance. For example, ANN models often get trapped into local minima and suffer from overfitting problems, while SVM models occasionally encounter overfitting problem [43]; moreover, some fuzzy information is not handled well by the standard SVM models.(f)
LS-SVM, FSVM, and LS-FSVM models outperform the other four models listed in this study, implying that the variants of SVM have a strong classification potentiality for credit risk evaluation. The possible reason is that the improvement of SVM variants overcomes some inherent limitations of standard SVM proposed by Van Gestel [44], thereby increasing the generalization capability. From a general point of view, LS-FSVM outperforms the other six classifiers from the above three measurements, revealing that the LS-FSVM is used as an effective tool for credit risk evaluation.In terms of Table3 and the three measurements, it is easy to judge which model is the best and which model is the worst. However, it is unclear what the differences between good models and bad ones are. For this purpose, McNemar’s test [45] is conducted to examine whether the proposed LS-FSVM classifier significantly outperforms the other six classifiers listed in this study.As a nonparametric test for the two related samples, McNemar’s test is particularly useful for before-after measurement of the same subjects [46]. Taking the total accuracy results from Table 3, Table 4 shows the results of the McNemar’s test for the credit dataset to statistically compare the performance in respect of testing data among the seven classifiers. It should be noted that the results listed in Table 4 are the Chi-squared values, and P values are in brackets.Table 4
McNemar’s test for pairwise performance comparison of different models.
Model
LS-SVM
FSVM
SVM
ANN
LogR
LinR
LS-FSVM
0.0000 [1.0000]
0.0480 [0.8262]
15.488 [0.0001]
26.881 [0.0001]
59.274 [0.0001]
92.160 [0.0001]
LS-SVM
0.0120 [0.9131]
14.675 [0.0001]
25.840 [0.0001]
57.840 [0.0001]
90.482 [0.0001]
FSVM
13.133 [0.0003]
23.842 [0.0001]
55.048 [0.0001]
87.197 [0.0001]
SVM
1.5370 [0.2501]
15.584 [0.0001]
36.300 [0.0001]
ANN
7.0840 [0.0078]
22.781 [0.0001]
LogR
4.3630 [0.0367]According to the results reported in Table4, some important conclusions can be drawn in terms of McNemar’s statistical test.(1)
The proposed LS-FSVM classifier outperforms the standard SVM, ANN, logit regression (LogR), and linear regression (LinR) models at 1% statistical significance level. However, the proposed LS-FSVM model does not significantly outperform the LS-SVM model and the FSVM model. These results are consistent with those of Table3.(2)
Similar to the LS-FSVM model, LS-SVM and FSVM models outperform the other four individual models (i.e., individual SVM, ANN, LogR, and LinR models) at 1% significance level. But the McNemar’s test does not conclude that the LS-SVM model performs better than the FSVM model.(3)
For SVM and ANN models, we can find that these two models perform much better than the two statistical models (i.e., LogR and LinR models) at 1% significance level. Interestingly, the SVM model does not outperform the ANN model at 10% significance level, although many applications have reported that the performance of the SVM was much better than that of ANN. The possible reason lies in data samples used in this study.(4)
Comparing with LogR and LinR models, it is easy to find that the LogR model performs better than the LinR model at 5% significance level. All findings are consistent with results reported in Table3.In summary, according to the above experimental results and statistical testing, it is easy to conclude that the LS-FSVM model can significantly outperform some standard intelligent models (e.g., SVM and ANN) and some statistical models (e.g., LogR and LinR), revealing that the LS-FSVM can also be used as a competitive solution to credit risk evaluation.


## 4. Conclusions

In this paper, a powerful classification method—least squares fuzzy support vector machines (LS-FSVM)—is proposed to evaluate credit risks. Through the least squares method, a quadratic programming (QP) problem of SVM can be transformed into a set of linear equations successfully, thereby reducing the computational complexity. Furthermore, the fuzzification processing in the proposed LS-FSVM model adds generalization capability and insensitivity to outliers. Experiments with real-world dataset have produced good classification results and fast computational efficiency and have demonstrated that the proposed LS-FSVM model can provide a feasible alternative to credit risk assessment. Besides the credit risk evaluation problem, the proposed LS-FSVM model can also be extended to other applications, such as consumer credit rating and corporate failure prediction problems, which will be investigated in the future research.



---

*Source: 564213-2014-06-12.xml*