# A Novel Ensemble Credit Scoring Model Based on Extreme Learning Machine and Generalized Fuzzy Soft Sets

Authors: Dayu Xu; Xuyao Zhang; Junguo Hu; Jiahao Chen
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

This paper mainly discusses the hybrid application of ensemble learning, classification, and feature selection (FS) algorithms simultaneously based on training data balancing for helping the proposed credit scoring model perform more effectively, which comprises three major stages. Firstly, it conducts preprocessing for collected credit data. Then, an efficient feature selection algorithm based on adaptive elastic net is employed to reduce the weakly related or uncorrelated variables to get high-quality training data. Thirdly, a novel ensemble strategy is proposed to make the imbalanced training data set balanced for each extreme learning machine (ELM) classifier. Finally, a new weighting method for single ELM classifiers in the ensemble model is established with respect to their classification accuracy based on generalized fuzzy soft sets (GFSS) theory. A novel cosine-based distance measurement algorithm of GFSS is also proposed to calculate the weights of each ELM classifier. To confirm the efficiency of the proposed ensemble credit scoring model, we implemented experiments with real-world credit data sets for comparison. The process of analysis, outcomes, and mathematical tests proved that the proposed model is capable of improving the effectiveness of classification in average accuracy, area under the curve (AUC), H-measure, and Brier’s score compared to all other single classifiers and ensemble approaches.


---

## Body

## 1. Introduction

Nowadays, financial institutions tend to adopt different risk assessment and credit scoring models to reduce potential risk to a certain extent [1]. By analyzing customer credit data to figure out the probability that potential borrowers will default on their loans, the evaluation approaches can be utilized to turn customer data into principle, which could support credit decisions [2]. In that way, an effective credit scoring model can be a reliable supporting system to help managers in making their financial decisions.To handle the potential risk of financial services, in the past few years, increasingly financial institutions are moving from traditional manual methods to advanced approaches that require building various types of evaluation models. For credit evaluation, three main methods, which are statistical approaches, nonparametric approaches, and AI methods, are being widely utilized [3–7]. These three methods work efficiently in various circumstances. Statistical methods consist of different models, which include discriminant analysis models, linear probability models, and probit and logit models. Yet nonparametric approaches tend to utilize the decision tree, K-nearest neighbor algorithm, fuzzy logic, Naïve Bayes, and so on. AI methods are more advanced and technology-dependent, such as artificial neural networks, support vector machines (SVM), particle swarm optimization (PSO), and genetic algorithm (GA).Many researches also indicated that ensemble approaches show more effective performance in the evaluation of credit than single classifiers. To avoid the downsides of single classifiers, an increasing number of researchers have switched to using customized and combined various methods instead of using individual classification models separately. The principle of the hybrid approach is to perform preprocessing on the data input to the classifiers. The focus is to gather information from group-based classifiers based on the same issue and then export these strengths to get valid credit scoring decisions [8, 9]. In recent years, the research of fuzzy soft sets theory has made great progress, especially in the application field of multiattribute decision making [10, 11]. The development of fuzzy soft sets theory can provide a new perspective for us to build more state-of-the-art ensemble data classification and credit evaluation models [12].The motivation of our study is to construct a more reliable credit scoring model that can generate accurate outcomes within imbalanced data. Three main approaches will be addressed to achieve this goal: (1) improved elastic net-based feature selection, (2) novel ensemble strategy and learning algorithm for imbalanced credit data, and (3) dynamic weighting method for single ELM classifiers based on new proposed similarity measure of GFSS. Because in real applications of credit risk evaluation, especially in peer-to-peer lending, credit data could be gathered from many different channels, including social networking and judicial administration platforms. The data collected from these channels are usually very sparse, redundant, rough, and imbalanced (good customers generally outnumber bad customers) and often consist of various weakly related or even uncorrelated features [13, 14]. These data characteristics will make the commonly used credit scoring models unstable, which leads to the credit evaluation results become unreliable and inaccurate. Through the above three approaches proposed in this article, the problems arising in credit scoring for imbalanced data can be handled and solved effectively.In Section2, we will talk about the construction of a new ensemble credit scoring model. The experimental outcomes will be discussed in Section 3. Finally, Section 4 concludes the paper.


## 2. New Ensemble Credit Scoring Model

This section mainly talks about the construction of the ensemble classification model for credit scoring.


### 2.1. Adaptive Elastic Net-Based Feature Selection

A large number of researchers have studied the appropriate feature selection approaches for credit scoring, such as cost-sensitive [15], information gain ratio [16], and genetic algorithm [17]. The Lasso estimator can reduce the regression coefficients to zero in L_1-norm. This method can also reduce features (variables) as well as select the most important one to build simple but effective models while keeping the high efficiency. Denote historical credit scoring data as xi,yi, i=1,2,…,N, where xi=xi1,xi2,…,xip are variables for customers and yi are category tags (binary responses, denote 0 as default and 1 as nondefault). The regression model could be constructed as follows:(1)yi=β0+∑j=1pβjxij,where β0 and βj are the intercept and regression coefficients, respectively. Suppose that every observation is not correlated and that all the variables are normalized. The Lasso proper estimate of β^ could be constructed as(2)β^=argmin∑i=1Nyi−∑j=1pβjxij2+λ∑j=1pβj.Based on the information above, a largeλ would reduce some coefficients in βj to zero. That is, Lasso reduces the coefficients to zero while λ gradually increases. In addition, the Lasso model is able to hold any number of variables. Therefore, both the reduction of coefficients and the selection of features (variables) can be carried out at the same time.Although Lasso has been proved to be easily interpretable and effective under various circumstances, it still has some shortcomings [18]. Zou and Hastie [19] put forward an expansion approach called elastic net to conduct selection. Similarly, the elastic net is also able to conduct automatic selection of variables and shrinkage of coefficient at the same time and select groups of correlated variables. For any constant and nonnegative λ1 and λ2, the estimation of β^ by the elastic net β^Enet could be carried out as follows:(3)β^Enet=argmin∑i=1Nyi−∑j=1pβjxij2+λ1∑j=1pβj+λ2∑j=1pβj2,where ∑j=1pβj is an element of L1-norm and λ2∑j=1pβj2 is the L2-norm element.In addition, Zou and Zhang [20] also pointed out that the elastic net does not possess the oracle property. They then proposed a new adaptive elastic net which combines the L2 penalty with the weighted L1 penalty to penalize the squared error loss. Therefore, the adaptive elastic net could be treated as the package of the adaptive Lasso and elastic net. The valuation of β^ by the adaptive elastic net β^Enet could be calculated as(4)β^Enet=argmin∑i=1Nyi−∑j=1pβjxij2+λ1∗∑j=1pω^jβj+λ2∑j=1pβj2,where ω^j=β^Enet−γ,γ is positive, while λ1∗ is fixed and nonnegative.Using formula (4), we will be capable of obtaining the most significant attributes (“big fish”) from the variable pool. Then, we can plug them into credit scoring models to get a more precise result but with minimum computational and operational cost.


### 2.2. ELM-Based Classifier

ELM model, as a single-hidden layer feedforward neural network (SLFN), can select the input weight and hidden biases randomly without any adjustment during its process. The Moore–Penrose generalized inverse matrices of the hidden-layer output matrix can be utilized to analyze and determine the output weights. ELM exhibits excellent performance of generalizing and the reduction in the iterative time of training process. Clearly, it is more effective than any other ANN-type machine learning algorithms [21].For the historical training credit data setxi,yi that is mentioned above, the input vector xi=xi1,xi2,…,xipT∈Rp is the ith sample with p-dimensional features, and Y=y1,y2,…,yN.Then, p is the amount of input neurons. p is also equivalent to the input features. Let L be the amount of hidden neurons. Denote C as the amount of output neurons, which is also equivalent to the category number. Denote the input weight matrix as Κ=k1,k2,…,kL, where kj=kj1,kj2,…,kjn is the vector connecting the p input neurons with the jth hidden neuron. b=b1,b2,…,bj,…,bLT is the bias value of the hidden neurons, where bj is the bias value of the jth hidden neurons. The above parameters do not change during the whole process. The output could be computed by as follows:(5)hxi=GΚxi+b,where G(x) is the activation function. Let H be the output of all the samples. It can be calculated by using the following equation:(6)H=h1x1h2x1⋯hLx1h1x2h2x2⋯hLx2⋮⋮⋯⋮h1xNh2xN⋯hLxNN×L.Theith column represents the ith hidden nodes output vector relative to the inputs x1 to xN. The jth row represents the output vector of the hidden layer relative to the input xj. The output of ELM can be calculated by(7)fx=∑i=1LαiGΚxi+b=oj,j=1,2,…,N,where αi=αi1,αi2,…,αiCT is the weight vector that connects the ith hidden nodes with the output nodes. ELM is able to evaluate those N samples without any mistake. In other words, ∑j=1Loj−yj=0. Then, the following equation can be obtained:(8)∑i=1LαiGKxi+b=yj,j=1,2,…,N.Equation (8) can also be rewritten as follows:(9)Hα=Y.Based on (9), the value of output weight could be estimated using a least square solution as follows:(10)α^=H†Y,where H† stands for the Moore–Penrose generalized inverse of H. For credit scoring classification, the outcome of ELM is as follows:(11)fx=sign∑i=1Lα^iGΚxi+b=signα^⋅hx.


### 2.3. Ensemble Strategy for Imbalanced Data

To better solve the classification of imbalanced data, a considerable number of approaches have been used. They can be categorized into three types: preprocessing, cost-sensitive learning, and ensemble methodology. Preprocessing is able to decrease the classification bias based on the bias-variance decomposition to enhance the single classifier. Undersampling [22, 23], oversampling [24, 25], and strategic sampling are extensively utilized to offset imbalanced data.Ensemble methodology can be viewed as a decision-making process that combines both individual learning algorithms and their outcomes in parallel to obtain the ultimate result. The basic idea behind the ensemble methodology is that the algorithm will get a number of single classifiers from the training set, and then it uses some ensemble strategies to integrate them to raise the accuracy and reliability of classification. Bagging [26], boosting [27], and stacking [28] are the most common ensemble approaches in credit scoring.A novel ensemble strategy is planned for imbalanced data according to its imbalance ratio, which could determine the number of ELM classifiers that apply as single classifiers to predict credit scoring data, as well as the number of samples that feed into each ELM as training data.For any given historical credit training data set that containsN samples, there are N+“good applicants” and N−“bad applicants,” such that N++N−=N. Then, the imbalance ratio is called IR, which can be calculated as follows:(12)IR=N+N−.After obtaining the IR, the amount of single ELM classifiersM in the ensemble model also can be calculated as follows:(13)M=IR=N+N−,where the symbol ⌈⋅⌉ represents “ceiling” operation.Equation (13) can help us to not only determine the number of ELM classifiers needed in the ensemble credit scoring model but will also guide us to make the imbalanced data become balanced for each classifier. Regarding the ensemble strategy for imbalanced data, we proposed M-based value. In the remainder of this subsection, we will elaborate the proposed strategy in detail.Firstly, calculate the imbalance ratioIR for any given historical credit training data set on the basis of N+ and N−. Secondly, determine the number of ELM classifiers M using (13).Thirdly, for the firstM-1 ELM classifiers, we feed N−“good applicants” samples and N−“bad applicants” samples into the ELM classifiers to make sure that the training data sets of the first M − 1 classifiers are balanced, and random sampling without replacement method is employed to extract N− samples from N+“good applicants” samples for each classifier. After finishing the first M-1 training data sets extraction, there are N+−M−1⋅N− “good applicants” sample that have not been extracted.Finally, for the last ELM classifier, we let the remainingN+−M−1⋅N−“good applicants” samples into the training data sets. Considering that N+−M−1⋅N−<N−, we will employ the SMOTE algorithm to create M⋅N−−N+“good applicants” samples from N+“good applicants” samples. Thus, for the last ELM classifier, there are still N−“good applicants” samples and N−“bad applicants” samples fed into it as the training data set.Through the described processes above, the ensemble strategy for the imbalanced data we proposed has been realized, and the training data for each classifier is balanced. In the next subsection, we will further introduce the GFSS theory-based ensemble credit scoring approach, which utilizes the results of each single ELM classifier.


### 2.4. GFSS Theory-Based Ensemble Credit Scoring Model

Since we have the results of each ELM classifier, we also need to figure out the weights with respect to their performance. The accuracy of classification is expected to be greatly improved. The theory of soft sets, which is firstly put forward by Molodtsov [29], can be regarded as a way for solving the uncertainties in imprecise environments (e.g., credit scoring area). Maji et al. [30] launched a research focusing on both fuzzy and soft sets. We will firstly introduce the principle of generalized fuzzy soft sets and then put forward a similarity measure of generalized fuzzy soft sets using angular cosine. After that, we can get the weights of each credit scoring model using similarity measure and the accuracy of classification. Finally, we are able to build the generalized fuzzy soft sets theory-based ensemble credit scoring model.


#### 2.4.1. GFSS Theory

Based on the theories proposed by Molodtsov [29] and Maji et al. [30], we can make some definitions for fuzzy soft sets.Definition 1.
DenoteU as the initial universal set. Denote P as a set of parameters. P(U) is the power set of U. (F, P) is a soft set over U if P is a mapping given by F : P ⟶ P(U).Definition 2.
DenoteU as the initial universal set. Denote P as a set of parameters. The power set of all fuzzy subsets of U is IU. Let A⊂P : A pair (F, P) be the fuzzy soft set over U if F is a mapping given by F : A ⟶ IU.
Then, Maji and Samanta’s [31] definition of GFSS is as follows:Definition 3.
DenoteU as the initial universal set. Denote P as a set of parameters. Let (U, P) be the soft universe. Denote F : P ⟶ IU and µ as the fuzzy subset of P, i.e., µ: P ⟶ I = [0, 1], where IU is all fuzzy subsets of U. Denote Fµ as the mapping given by Fµ : P ⟶ IU × I, which can also be denoted as Fµ(e) = (F(e), µ(e)), where F(e) ∈ IU. In this way, Fµ can be viewed as a GFSS over the soft universe (U, P).
For everyei, Fµ (ei) = (F(ei), µ(ei)) illustrates both the level of belonging of the subsets of U to F(ei) and the possibility of belonging.
In this paper,U denotes the historical data of customer credit X=x1,x2,…,xn, where xi∈Rp, and F(ei) represents the performance of the classification by a single customer from a single classifier and µ(ei) denotes the overall degree of classification of a certain single classifier.


#### 2.4.2. Similarity Measure of GFSS

It is important to address the similarity measurement of GFSS during the setting of GFSS and the establishment of our model. Thus, we establish a new approach for the similarity measure of GFSS for the scoring of credits.Definition 4.
ForM single classifiers, denote ηt and μm as the elements of Fµ. The definition is as follows:(14)ηtm=1−y^tm−yt1+y^tm−yt,where ytt=1,2,…,T is the category tag of the tth customer (binary responses denote 0 as default and 1 as nondefault); y^tmm=1,2,…,M is the forecasting result of the tth customer predicted by the mth classifier and is between 0 and 1. ηtm is the accuracy degree of classification of a single classifier for a single customer. This accuracy ranges between 0 and 1. This is in line with our initial intuition. In addition, μm is the mth classifier’s overall classification performance, which can be calculated as follows:(15)μm=TPmTPm+FNm⋅TNmTNm+FPm,where TPm, FNm, TNm, and FPm are elements of the confusion matrix (Table 1). TPm is the amount of good customers accurately labeled as good, TNm is the amount of bad customers accurately labeled as bad, FNm is the amount of good customers falsely labeled as bad, and FPm is the amount of bad customers falsely labeled as good. The greater the value of μm, the more accurate the result is calculated by the mth classifier.
Based on the above discussion, it can be noted thatηit and μm from Definition 4 are able to evaluate the performance of classification of every single model. Therefore, we can build the GFSS Fμmη,μ of the mth classifier as follows:(16)Fμmη,μ=y1ηtm,y2η2m,…,ytηtm,…,yTηTm,μm.Table 1
Confusion matrix.
Real class (%)Projected class (%)GoodBadGoodTPmFNmBadFPmTNmDefinition 5.
For two GFSSFμi and Fμj over the universal set U, where i,j∈1,M, the similarity measurement of GFSS SGFμi,Fμj can be calculated as follows:(17)SGFμi,Fμj=∑t=1Tηtiμi×ηtjμj∑t=1Tηtiμi2∑t=1Tηtjμj2.


#### 2.4.3. Ensemble Credit Scoring Modeling

The determination of the weight for every single model is the most important step in the establishment of the ensemble credit score modeling. The purpose of this determination is to compare the calculated credit score with the actual records.The similarity measure of GFSS can be utilized to do the determination, i.e., the calculatedSGFμm,Fμ∗ value, where Fμ∗ is the GFSS of the actual score of customers. In this way, the weight of mth classifier can be calculated by(18)ϖm=SGFμm,Fμ∗∑m=1MSGFμm,Fμ∗,where 0≤ϖm≤1 and ∑m=1Mϖm=1. Thus, the final score could be calculated as(19)y^t=∑m=1Mϖmy^tm.Figure1 presents the flow-process diagram of the ensemble model. The algorithm of “ELM and GFSS Theory-Based Hybrid Ensemble” model is described. We will call this EGHE in the following parts (Algorithm 1).Figure 1
Flow diagram of the proposed EGHE model.Algorithm 1:EGHE algorithm.
Input: Historical data of credit scoring (xi, yi).Output: Score of every single customer y^t.Step 1. Preprocessing of data.Step 2. Variables selection using AEnet.Step 3. Imbalanced data rebalancing by using the proposed ensemble strategy.Step 4. Credit scoring of every single ELM classifier.Step 5. Computeηt and μm using (14) and (15).Step 6. Calculate theSGFμm,Fμ∗ of single classifier using (17).Step 7. Calculate the weight ofmth classifier using (18).Step 8. Get the final credit score of every customery^t.


## 2.1. Adaptive Elastic Net-Based Feature Selection

A large number of researchers have studied the appropriate feature selection approaches for credit scoring, such as cost-sensitive [15], information gain ratio [16], and genetic algorithm [17]. The Lasso estimator can reduce the regression coefficients to zero in L_1-norm. This method can also reduce features (variables) as well as select the most important one to build simple but effective models while keeping the high efficiency. Denote historical credit scoring data as xi,yi, i=1,2,…,N, where xi=xi1,xi2,…,xip are variables for customers and yi are category tags (binary responses, denote 0 as default and 1 as nondefault). The regression model could be constructed as follows:(1)yi=β0+∑j=1pβjxij,where β0 and βj are the intercept and regression coefficients, respectively. Suppose that every observation is not correlated and that all the variables are normalized. The Lasso proper estimate of β^ could be constructed as(2)β^=argmin∑i=1Nyi−∑j=1pβjxij2+λ∑j=1pβj.Based on the information above, a largeλ would reduce some coefficients in βj to zero. That is, Lasso reduces the coefficients to zero while λ gradually increases. In addition, the Lasso model is able to hold any number of variables. Therefore, both the reduction of coefficients and the selection of features (variables) can be carried out at the same time.Although Lasso has been proved to be easily interpretable and effective under various circumstances, it still has some shortcomings [18]. Zou and Hastie [19] put forward an expansion approach called elastic net to conduct selection. Similarly, the elastic net is also able to conduct automatic selection of variables and shrinkage of coefficient at the same time and select groups of correlated variables. For any constant and nonnegative λ1 and λ2, the estimation of β^ by the elastic net β^Enet could be carried out as follows:(3)β^Enet=argmin∑i=1Nyi−∑j=1pβjxij2+λ1∑j=1pβj+λ2∑j=1pβj2,where ∑j=1pβj is an element of L1-norm and λ2∑j=1pβj2 is the L2-norm element.In addition, Zou and Zhang [20] also pointed out that the elastic net does not possess the oracle property. They then proposed a new adaptive elastic net which combines the L2 penalty with the weighted L1 penalty to penalize the squared error loss. Therefore, the adaptive elastic net could be treated as the package of the adaptive Lasso and elastic net. The valuation of β^ by the adaptive elastic net β^Enet could be calculated as(4)β^Enet=argmin∑i=1Nyi−∑j=1pβjxij2+λ1∗∑j=1pω^jβj+λ2∑j=1pβj2,where ω^j=β^Enet−γ,γ is positive, while λ1∗ is fixed and nonnegative.Using formula (4), we will be capable of obtaining the most significant attributes (“big fish”) from the variable pool. Then, we can plug them into credit scoring models to get a more precise result but with minimum computational and operational cost.


## 2.2. ELM-Based Classifier

ELM model, as a single-hidden layer feedforward neural network (SLFN), can select the input weight and hidden biases randomly without any adjustment during its process. The Moore–Penrose generalized inverse matrices of the hidden-layer output matrix can be utilized to analyze and determine the output weights. ELM exhibits excellent performance of generalizing and the reduction in the iterative time of training process. Clearly, it is more effective than any other ANN-type machine learning algorithms [21].For the historical training credit data setxi,yi that is mentioned above, the input vector xi=xi1,xi2,…,xipT∈Rp is the ith sample with p-dimensional features, and Y=y1,y2,…,yN.Then, p is the amount of input neurons. p is also equivalent to the input features. Let L be the amount of hidden neurons. Denote C as the amount of output neurons, which is also equivalent to the category number. Denote the input weight matrix as Κ=k1,k2,…,kL, where kj=kj1,kj2,…,kjn is the vector connecting the p input neurons with the jth hidden neuron. b=b1,b2,…,bj,…,bLT is the bias value of the hidden neurons, where bj is the bias value of the jth hidden neurons. The above parameters do not change during the whole process. The output could be computed by as follows:(5)hxi=GΚxi+b,where G(x) is the activation function. Let H be the output of all the samples. It can be calculated by using the following equation:(6)H=h1x1h2x1⋯hLx1h1x2h2x2⋯hLx2⋮⋮⋯⋮h1xNh2xN⋯hLxNN×L.Theith column represents the ith hidden nodes output vector relative to the inputs x1 to xN. The jth row represents the output vector of the hidden layer relative to the input xj. The output of ELM can be calculated by(7)fx=∑i=1LαiGΚxi+b=oj,j=1,2,…,N,where αi=αi1,αi2,…,αiCT is the weight vector that connects the ith hidden nodes with the output nodes. ELM is able to evaluate those N samples without any mistake. In other words, ∑j=1Loj−yj=0. Then, the following equation can be obtained:(8)∑i=1LαiGKxi+b=yj,j=1,2,…,N.Equation (8) can also be rewritten as follows:(9)Hα=Y.Based on (9), the value of output weight could be estimated using a least square solution as follows:(10)α^=H†Y,where H† stands for the Moore–Penrose generalized inverse of H. For credit scoring classification, the outcome of ELM is as follows:(11)fx=sign∑i=1Lα^iGΚxi+b=signα^⋅hx.


## 2.3. Ensemble Strategy for Imbalanced Data

To better solve the classification of imbalanced data, a considerable number of approaches have been used. They can be categorized into three types: preprocessing, cost-sensitive learning, and ensemble methodology. Preprocessing is able to decrease the classification bias based on the bias-variance decomposition to enhance the single classifier. Undersampling [22, 23], oversampling [24, 25], and strategic sampling are extensively utilized to offset imbalanced data.Ensemble methodology can be viewed as a decision-making process that combines both individual learning algorithms and their outcomes in parallel to obtain the ultimate result. The basic idea behind the ensemble methodology is that the algorithm will get a number of single classifiers from the training set, and then it uses some ensemble strategies to integrate them to raise the accuracy and reliability of classification. Bagging [26], boosting [27], and stacking [28] are the most common ensemble approaches in credit scoring.A novel ensemble strategy is planned for imbalanced data according to its imbalance ratio, which could determine the number of ELM classifiers that apply as single classifiers to predict credit scoring data, as well as the number of samples that feed into each ELM as training data.For any given historical credit training data set that containsN samples, there are N+“good applicants” and N−“bad applicants,” such that N++N−=N. Then, the imbalance ratio is called IR, which can be calculated as follows:(12)IR=N+N−.After obtaining the IR, the amount of single ELM classifiersM in the ensemble model also can be calculated as follows:(13)M=IR=N+N−,where the symbol ⌈⋅⌉ represents “ceiling” operation.Equation (13) can help us to not only determine the number of ELM classifiers needed in the ensemble credit scoring model but will also guide us to make the imbalanced data become balanced for each classifier. Regarding the ensemble strategy for imbalanced data, we proposed M-based value. In the remainder of this subsection, we will elaborate the proposed strategy in detail.Firstly, calculate the imbalance ratioIR for any given historical credit training data set on the basis of N+ and N−. Secondly, determine the number of ELM classifiers M using (13).Thirdly, for the firstM-1 ELM classifiers, we feed N−“good applicants” samples and N−“bad applicants” samples into the ELM classifiers to make sure that the training data sets of the first M − 1 classifiers are balanced, and random sampling without replacement method is employed to extract N− samples from N+“good applicants” samples for each classifier. After finishing the first M-1 training data sets extraction, there are N+−M−1⋅N− “good applicants” sample that have not been extracted.Finally, for the last ELM classifier, we let the remainingN+−M−1⋅N−“good applicants” samples into the training data sets. Considering that N+−M−1⋅N−<N−, we will employ the SMOTE algorithm to create M⋅N−−N+“good applicants” samples from N+“good applicants” samples. Thus, for the last ELM classifier, there are still N−“good applicants” samples and N−“bad applicants” samples fed into it as the training data set.Through the described processes above, the ensemble strategy for the imbalanced data we proposed has been realized, and the training data for each classifier is balanced. In the next subsection, we will further introduce the GFSS theory-based ensemble credit scoring approach, which utilizes the results of each single ELM classifier.


## 2.4. GFSS Theory-Based Ensemble Credit Scoring Model

Since we have the results of each ELM classifier, we also need to figure out the weights with respect to their performance. The accuracy of classification is expected to be greatly improved. The theory of soft sets, which is firstly put forward by Molodtsov [29], can be regarded as a way for solving the uncertainties in imprecise environments (e.g., credit scoring area). Maji et al. [30] launched a research focusing on both fuzzy and soft sets. We will firstly introduce the principle of generalized fuzzy soft sets and then put forward a similarity measure of generalized fuzzy soft sets using angular cosine. After that, we can get the weights of each credit scoring model using similarity measure and the accuracy of classification. Finally, we are able to build the generalized fuzzy soft sets theory-based ensemble credit scoring model.


### 2.4.1. GFSS Theory

Based on the theories proposed by Molodtsov [29] and Maji et al. [30], we can make some definitions for fuzzy soft sets.Definition 1.
DenoteU as the initial universal set. Denote P as a set of parameters. P(U) is the power set of U. (F, P) is a soft set over U if P is a mapping given by F : P ⟶ P(U).Definition 2.
DenoteU as the initial universal set. Denote P as a set of parameters. The power set of all fuzzy subsets of U is IU. Let A⊂P : A pair (F, P) be the fuzzy soft set over U if F is a mapping given by F : A ⟶ IU.
Then, Maji and Samanta’s [31] definition of GFSS is as follows:Definition 3.
DenoteU as the initial universal set. Denote P as a set of parameters. Let (U, P) be the soft universe. Denote F : P ⟶ IU and µ as the fuzzy subset of P, i.e., µ: P ⟶ I = [0, 1], where IU is all fuzzy subsets of U. Denote Fµ as the mapping given by Fµ : P ⟶ IU × I, which can also be denoted as Fµ(e) = (F(e), µ(e)), where F(e) ∈ IU. In this way, Fµ can be viewed as a GFSS over the soft universe (U, P).
For everyei, Fµ (ei) = (F(ei), µ(ei)) illustrates both the level of belonging of the subsets of U to F(ei) and the possibility of belonging.
In this paper,U denotes the historical data of customer credit X=x1,x2,…,xn, where xi∈Rp, and F(ei) represents the performance of the classification by a single customer from a single classifier and µ(ei) denotes the overall degree of classification of a certain single classifier.


### 2.4.2. Similarity Measure of GFSS

It is important to address the similarity measurement of GFSS during the setting of GFSS and the establishment of our model. Thus, we establish a new approach for the similarity measure of GFSS for the scoring of credits.Definition 4.
ForM single classifiers, denote ηt and μm as the elements of Fµ. The definition is as follows:(14)ηtm=1−y^tm−yt1+y^tm−yt,where ytt=1,2,…,T is the category tag of the tth customer (binary responses denote 0 as default and 1 as nondefault); y^tmm=1,2,…,M is the forecasting result of the tth customer predicted by the mth classifier and is between 0 and 1. ηtm is the accuracy degree of classification of a single classifier for a single customer. This accuracy ranges between 0 and 1. This is in line with our initial intuition. In addition, μm is the mth classifier’s overall classification performance, which can be calculated as follows:(15)μm=TPmTPm+FNm⋅TNmTNm+FPm,where TPm, FNm, TNm, and FPm are elements of the confusion matrix (Table 1). TPm is the amount of good customers accurately labeled as good, TNm is the amount of bad customers accurately labeled as bad, FNm is the amount of good customers falsely labeled as bad, and FPm is the amount of bad customers falsely labeled as good. The greater the value of μm, the more accurate the result is calculated by the mth classifier.
Based on the above discussion, it can be noted thatηit and μm from Definition 4 are able to evaluate the performance of classification of every single model. Therefore, we can build the GFSS Fμmη,μ of the mth classifier as follows:(16)Fμmη,μ=y1ηtm,y2η2m,…,ytηtm,…,yTηTm,μm.Table 1
Confusion matrix.
Real class (%)Projected class (%)GoodBadGoodTPmFNmBadFPmTNmDefinition 5.
For two GFSSFμi and Fμj over the universal set U, where i,j∈1,M, the similarity measurement of GFSS SGFμi,Fμj can be calculated as follows:(17)SGFμi,Fμj=∑t=1Tηtiμi×ηtjμj∑t=1Tηtiμi2∑t=1Tηtjμj2.


### 2.4.3. Ensemble Credit Scoring Modeling

The determination of the weight for every single model is the most important step in the establishment of the ensemble credit score modeling. The purpose of this determination is to compare the calculated credit score with the actual records.The similarity measure of GFSS can be utilized to do the determination, i.e., the calculatedSGFμm,Fμ∗ value, where Fμ∗ is the GFSS of the actual score of customers. In this way, the weight of mth classifier can be calculated by(18)ϖm=SGFμm,Fμ∗∑m=1MSGFμm,Fμ∗,where 0≤ϖm≤1 and ∑m=1Mϖm=1. Thus, the final score could be calculated as(19)y^t=∑m=1Mϖmy^tm.Figure1 presents the flow-process diagram of the ensemble model. The algorithm of “ELM and GFSS Theory-Based Hybrid Ensemble” model is described. We will call this EGHE in the following parts (Algorithm 1).Figure 1
Flow diagram of the proposed EGHE model.Algorithm 1:EGHE algorithm.
Input: Historical data of credit scoring (xi, yi).Output: Score of every single customer y^t.Step 1. Preprocessing of data.Step 2. Variables selection using AEnet.Step 3. Imbalanced data rebalancing by using the proposed ensemble strategy.Step 4. Credit scoring of every single ELM classifier.Step 5. Computeηt and μm using (14) and (15).Step 6. Calculate theSGFμm,Fμ∗ of single classifier using (17).Step 7. Calculate the weight ofmth classifier using (18).Step 8. Get the final credit score of every customery^t.


## 2.4.1. GFSS Theory

Based on the theories proposed by Molodtsov [29] and Maji et al. [30], we can make some definitions for fuzzy soft sets.Definition 1.
DenoteU as the initial universal set. Denote P as a set of parameters. P(U) is the power set of U. (F, P) is a soft set over U if P is a mapping given by F : P ⟶ P(U).Definition 2.
DenoteU as the initial universal set. Denote P as a set of parameters. The power set of all fuzzy subsets of U is IU. Let A⊂P : A pair (F, P) be the fuzzy soft set over U if F is a mapping given by F : A ⟶ IU.
Then, Maji and Samanta’s [31] definition of GFSS is as follows:Definition 3.
DenoteU as the initial universal set. Denote P as a set of parameters. Let (U, P) be the soft universe. Denote F : P ⟶ IU and µ as the fuzzy subset of P, i.e., µ: P ⟶ I = [0, 1], where IU is all fuzzy subsets of U. Denote Fµ as the mapping given by Fµ : P ⟶ IU × I, which can also be denoted as Fµ(e) = (F(e), µ(e)), where F(e) ∈ IU. In this way, Fµ can be viewed as a GFSS over the soft universe (U, P).
For everyei, Fµ (ei) = (F(ei), µ(ei)) illustrates both the level of belonging of the subsets of U to F(ei) and the possibility of belonging.
In this paper,U denotes the historical data of customer credit X=x1,x2,…,xn, where xi∈Rp, and F(ei) represents the performance of the classification by a single customer from a single classifier and µ(ei) denotes the overall degree of classification of a certain single classifier.


## 2.4.2. Similarity Measure of GFSS

It is important to address the similarity measurement of GFSS during the setting of GFSS and the establishment of our model. Thus, we establish a new approach for the similarity measure of GFSS for the scoring of credits.Definition 4.
ForM single classifiers, denote ηt and μm as the elements of Fµ. The definition is as follows:(14)ηtm=1−y^tm−yt1+y^tm−yt,where ytt=1,2,…,T is the category tag of the tth customer (binary responses denote 0 as default and 1 as nondefault); y^tmm=1,2,…,M is the forecasting result of the tth customer predicted by the mth classifier and is between 0 and 1. ηtm is the accuracy degree of classification of a single classifier for a single customer. This accuracy ranges between 0 and 1. This is in line with our initial intuition. In addition, μm is the mth classifier’s overall classification performance, which can be calculated as follows:(15)μm=TPmTPm+FNm⋅TNmTNm+FPm,where TPm, FNm, TNm, and FPm are elements of the confusion matrix (Table 1). TPm is the amount of good customers accurately labeled as good, TNm is the amount of bad customers accurately labeled as bad, FNm is the amount of good customers falsely labeled as bad, and FPm is the amount of bad customers falsely labeled as good. The greater the value of μm, the more accurate the result is calculated by the mth classifier.
Based on the above discussion, it can be noted thatηit and μm from Definition 4 are able to evaluate the performance of classification of every single model. Therefore, we can build the GFSS Fμmη,μ of the mth classifier as follows:(16)Fμmη,μ=y1ηtm,y2η2m,…,ytηtm,…,yTηTm,μm.Table 1
Confusion matrix.
Real class (%)Projected class (%)GoodBadGoodTPmFNmBadFPmTNmDefinition 5.
For two GFSSFμi and Fμj over the universal set U, where i,j∈1,M, the similarity measurement of GFSS SGFμi,Fμj can be calculated as follows:(17)SGFμi,Fμj=∑t=1Tηtiμi×ηtjμj∑t=1Tηtiμi2∑t=1Tηtjμj2.


## 2.4.3. Ensemble Credit Scoring Modeling

The determination of the weight for every single model is the most important step in the establishment of the ensemble credit score modeling. The purpose of this determination is to compare the calculated credit score with the actual records.The similarity measure of GFSS can be utilized to do the determination, i.e., the calculatedSGFμm,Fμ∗ value, where Fμ∗ is the GFSS of the actual score of customers. In this way, the weight of mth classifier can be calculated by(18)ϖm=SGFμm,Fμ∗∑m=1MSGFμm,Fμ∗,where 0≤ϖm≤1 and ∑m=1Mϖm=1. Thus, the final score could be calculated as(19)y^t=∑m=1Mϖmy^tm.Figure1 presents the flow-process diagram of the ensemble model. The algorithm of “ELM and GFSS Theory-Based Hybrid Ensemble” model is described. We will call this EGHE in the following parts (Algorithm 1).Figure 1
Flow diagram of the proposed EGHE model.Algorithm 1:EGHE algorithm.
Input: Historical data of credit scoring (xi, yi).Output: Score of every single customer y^t.Step 1. Preprocessing of data.Step 2. Variables selection using AEnet.Step 3. Imbalanced data rebalancing by using the proposed ensemble strategy.Step 4. Credit scoring of every single ELM classifier.Step 5. Computeηt and μm using (14) and (15).Step 6. Calculate theSGFμm,Fμ∗ of single classifier using (17).Step 7. Calculate the weight ofmth classifier using (18).Step 8. Get the final credit score of every customery^t.


## 3. Results and Discussion

### 3.1. Preparation of Dataset

During the evaluation, we collected many different private and public data sets. We have collected a total of six credit data sets and four additional imbalanced data sets with different IR are obtained, i.e., three public and three private. The public sets can be obtained from the UCI Machine Learning Repository. They are real-world credit score data sets and are now widely used by researchers. The German, Australian, and Japanese data sets are used for extra verification. The private data sets consist of the Iranian data set that has also been widely used in many studies and the Bene 1 and 2 data sets, which can be obtained from two key financial institutions in Benelux et al. [32]. This Iranian set has various customer data of many small Iranian private banks [33, 34]. Four additional imbalanced datasets are also from the Machine Learning Repository, UCI. They are Shuttle, Skin_segment, MiniBooNE, and LC2017Q1 which contains loan data of the first quarter in 2017 from Lending Club. The characteristics of all experimental data sets can be found in Table 2.Table 2
Characteristics of data sets.
Data setsSizeAttributesGood/badImbalance ratioNumber of classifiersCredit scoring data setsGermany100025700/3002.333Australia69014307/3830.801Japan69015296/3570.831Iran100027950/501919Bene 13,123332,082/1,04122Bene 27,190335,033/2,1572.333Additional imbalanced data setsShuttle12,380911,428/95212.00413Skin_segment117,7283114,039/3,67930.9931MiniBooNE201,35510196,555/4,80040.9541LC2017Q195,6337294,414/1,21977.4578In this paper, we compared our proposed model with the other four state-of-the-art models, namely, C5.0 decision tree, SVM with Radial Basis Function, kernel SVM-R, Deep Belief Networks (DBN), and Bayes to validate the performance of our approaches. All the continuous attributes will be discretized into various intervals. Every single data set will be divided into a two-thirds training set and one-third testing set randomly. We use the open source platformR-statistics (version R-3.2.2.) to conduct our experiments.


### 3.2. Experimental Results

Different methods are utilized as comparison models to test the validity of the EGHE credit scoring model.Firstly, the FS algorithm that is based on AEnet is used to obtain the highly correlated variables after initial data gathering and preprocessing. We could notice that after selection, the variables in these ten data sets are all decreased to various degrees (Table3). In consideration of the complexity of computation, deleting irrelevant or weakly correlated variables is becoming increasingly important for big-data-oriented credit assessment issues.Table 3
Number of features before and after AEnet-based selection.
Data setsNumber of featuresBeforeAfterGermany2521Australia1410Japan1512Iran2722Bene 13315Bene 23320Shuttle1613Skin_segment33MiniBooNE109LC2017Q17248After feature selection, we can perform two experiments to further see the effect of feature selection on the classification of each model: (1) all single classifiers with feature selection and (2) all single classifiers without feature selection. Tables4 and 5 show in detail the AUC, H-measure (HM), Brier’s score (BS), and accuracy (ACC) for all single classifiers with and without feature selection.Table 4
Results for all single classifiers without AEnet-based feature selection.
Data setPerformance measuresSingle classifiersC5.0SVM-RDBNBayesELMGermanyAUC0.6800.7320.7530.7610.764HM0.3360.3620.3990.3380.379BS0.2510.1670.1730.1990.172ACC0.7450.7520.7490.7450.762AustraliaAUC0.8670.8120.8760.8570.852HM0.5160.6030.6130.5810.610BS0.1480.1130.1070.1660.127ACC0.7360.7420.7510.7340.735JapanAUC0.8570.9080.9110.8880.882HM0.6890.6080.6080.6670.618BS0.1560.1200.1380.1760.142ACC0.7380.7490.7570.7420.751IranAUC0.6160.6040.6140.7150.710HM0.1110.0740.0610.1920.133BS0.0710.0710.0780.0770.083ACC0.6230.6170.6400.6270.648Bene 1AUC0.7280.8210.7670.7400.752HM0.3100.3660.2500.3350.351BS0.2670.1760.1680.2970.199ACC0.7010.7490.6980.6900.753Bene 2AUC0.7450.7890.7460.7070.762HM0.3850.3090.2380.2760.270BS0.1430.1780.1420.1720.153ACC0.7280.7200.7150.7110.747ShuttleAUC0.6040.6750.6930.7140.722HM0.2990.3340.3670.3170.358BS0.2240.1530.1580.1860.163ACC0.6620.6930.7400.6980.720Skin_segmentAUC0.7700.7490.8040.8020.805HM0.4580.5560.5640.5430.576BS0.1310.1050.0990.1560.119ACC0.6530.6850.6890.6870.695MiniBooNEAUC0.7610.8370.8380.8330.833HM0.6130.5600.5590.6250.584BS0.1400.1120.1260.1640.134ACC0.6550.6900.6960.6950.710LC2017Q1AUC0.5470.5570.5630.6690.671HM0.0990.0670.0570.1810.126BS0.0620.0660.0720.0710.078ACC0.6550.6700.6880.6870.712Table 5
Results for all single classifiers with AEnet-based feature selection.
Data setPerformance measurementSingle classifiersC5.0SVM-RDBNBayesELMGermanyAUC0.6900.7560.7620.7740.768HM0.1810.2950.2480.2670.268BS0.2300.1630.1720.1920.173ACC0.7510.7590.7520.7580.765AustraliaAUC0.8820.9020.9150.9120.883HM0.6140.6350.6420.6230.624BS0.1210.1040.1310.1210.130ACC0.7490.7500.7550.7410.748JapanAUC0.8610.9100.9190.9100.878HM0.6080.6210.6310.6220.622BS0.1160.1120.1260.1130.119ACC0.7490.7520.7660.7530.758IranAUC0.6310.6500.6120.7230.638HM0.1370.1180.1080.2120.117BS0.0680.0710.0780.0720.081ACC0.6520.6450.6500.6520.658Bene 1AUC0.7700.8100.8060.7730.798HM0.3120.3450.3360.3020.320BS0.2330.1770.1840.2640.241ACC0.7210.7460.7190.7210.753Bene 2AUC0.7630.8240.8150.7800.796HM0.3540.3860.2690.2850.326BS0.1320.1190.1370.1250.137ACC0.7340.7340.7150.7230.747ShuttleAUC0.6520.6820.6860.7230.729HM0.3230.3370.3630.3210.362BS0.2420.1550.1560.1880.165ACC0.6850.6990.7430.7070.727Skin_segmentAUC0.8320.7560.7960.8120.813HM0.4950.5620.5580.5500.582BS0.1410.1060.0980.1580.120ACC0.6750.6920.7120.6960.702MiniBooNEAUC0.8220.8450.8300.8440.841HM0.6620.5660.5530.6330.589BS0.1510.1130.1250.1660.135ACC0.7070.7040.6890.7040.717LC2017Q1AUC0.5910.5630.5570.6780.678HM0.1070.0680.0560.1830.125BS0.0670.0670.0710.0720.079ACC0.6820.6870.6810.6960.719From the results of every single classifier in Tables4 and 5, we can see that the feature selection helps enhance the effectiveness of classification of most single classifiers. After feature selection, C5.0 shows accuracy values that are 0.8%, 1.76%, 1.63%, 4.49%, 2.85%, 0.82%, 3.47%, 4.15%, 7.94%, and 4.12% greater than without feature selection for Germany, Australia, Japan, Iran, Bene 1, Bene 2, Shuttle, Skin_segment, MiniBooNE, and LC2017Q1 data sets, respectively. In the same way, SVM-R increased the accuracy values by 0.93%, 1.08%, 0.67%, 4.53%, −0.4%, 1.94%, 0.87%, 1.02%, 2.03%, and 2.54%, respectively, for ten experimental data sets. DBN and Naïve Bayes improve their effectiveness too after feature selection. Only SVM-R on Bene 1 reduced by 0.4%, but this does not contradict the improvement on the classification that our AEnet-based feature selection brings. Not only do outliers, redundant, and weakly or even unrelated variables help improve the effectiveness, but also affect the model establishment and cause great computation.It is noteworthy that, compared with C5.0, SVM-R, DBN, and Bayes, ELM has manifested the superiority in accuracy on the vast majority of data sets. Table6 reports the average running time (total time for training and testing) for all models. For these experiments, we use an Intel i5-8500 with CPU at 3.0 GHz and 16 GB of RAM.Table 6
Running time for all models with feature selection (seconds).
Data setsModelsC5.0SVM-RDBNBayesELMGermany1.762.894.331.721.38Australia1.162.022.991.070.97Japan1.252.052.901.130.94Iran1.823.014.311.691.44Bene 15.6310.1114.295.854.62Bene 212.7121.1328.9612.3210.01Shuttle21.7935.7853.6121.2917.11Skin_segment207.20340.24509.76202.49182.46MiniBooNE354.38581.92871.87346.33277.87LC2017Q1591.26970.781254.49560.57436.04From Table6, we can see that ELM costs less time than other single models to carry out credit scoring activities. The efficiency of computing resources also makes ELM a great match for ensemble learning and modeling.After implementing feature selection, completing the ensemble strategy, and individual model classification, the EGHE model can be achieved. Based on (14), (15), and (17)–(19), the weights of single ELM classifiers are calculated according to their efficiency, respectively. To validate the availability of EGHE, we employed several ensemble models in contrast with EGHE. These models were split into two parts. The first one contains four FS algorithms with GFSS-based combination. They are cost-sensitive, GA, information gain ratio (IGR), and elastic net (Enet). Cost-sensitive, GA, and IGR are popular feature selection approaches in the credit scoring area [16, 35, 36]. The second group applies four other approaches with AEnet-based feature selection, which were weighted average (WAVG) [37], majority voting (MajVot) [38], weighted voting (WVOT) [39], and fuzzy soft set (FSS). Those methods are frequently adopted in the establishment and utilization of different combination models. They also employ ELM as the classifier but did not take the ensemble strategy that is proposed above, only using random sampling methods to make all training data sets become balanced. Table 7 displays the results of AUC, H-measure, and Brier’s score for all ensemble models.Table 7
Classification outcomes from all data sets for different performance measures with various combinations.
Data setPerformance measurementOther feature selection methods with GFSS-based combinationTraditional combination methods with AEnet-based feature selectionEGHECost-sensitiveGAIGREnetWAVGMajVotWVOTFSSGermanyAUC0.7860.7920.7770.7810.7770.7860.7890.7710.823HM0.2860.1770.2260.2380.2220.2850.2460.3050.325BS0.1660.2080.1970.1820.1810.1830.1920.1570.184ACC0.8420.8620.8330.8270.8450.8590.8360.8440.886AustraliaAUC0.9210.9270.9330.9280.9210.9220.9320.9280.945HM0.6670.5190.5730.6280.6370.6370.6320.6530.659BS0.0920.1450.1730.1030.1010.1120.1070.0970.095ACC0.8720.8780.8820.8630.8720.8730.8760.8740.895JapanAUC0.9230.9280.9180.9320.9180.9270.9280.9250.937HM0.6500.4920.5660.6180.6020.6420.6170.6480.660BS0.0990.1570.1740.1090.1210.1120.1160.1030.098ACC0.8710.8620.8610.8680.8530.8640.8730.8680.904IranAUC0.7910.7780.7810.7670.7770.7790.7830.7780.802HM0.2790.1500.2170.1620.2830.1080.1070.2940.303BS0.0420.0690.0560.0490.0440.0480.0490.0440.059ACC0.8830.8810.8740.8610.8670.8850.8770.8830.915Bene 1AUC0.8430.8120.8220.8210.8820.8790.8860.8880.881HM0.3960.2630.3380.2580.3240.4410.3850.4470.351BS0.1610.2400.2560.1970.1880.1590.1980.1540.173ACC0.8810.8720.8640.8720.8720.8720.8650.8760.896Bene 2AUC0.9210.8380.8580.8680.8780.8440.8760.8830.888HM0.5370.4010.4780.4890.5050.4360.4320.4960.506BS0.0910.1360.1680.1220.1020.1150.1120.1030.114ACC0.8780.8600.8570.8680.8810.8750.8820.8840.898ShuttleAUC0.8960.9140.9140.9220.8970.9190.9170.9060.943HM0.6530.5100.5620.6250.6200.6350.6220.6360.658BS0.0910.1430.1700.1030.0990.1100.1040.0960.095ACC0.8520.8660.8620.8580.8490.8700.8620.8510916Skin_segmentAUC0.9020.9130.8990.9260.8960.9220.9150.9030.936HM0.6370.4860.5550.6150.5860.6380.6070.6310.659BS0.0960.1540.1710.1070.1190.1100.1130.0980.098ACC0.8530.8490.8440.8630.8330.8620.8590.8450.908MiniBooNEAUC0.7730.7670.7640.7630.7570.7750.7700.7580.800HM0.2720.1490.2130.1590.2770.1090.1040.2860.302BS0.0420.0670.0540.0480.04390.0470.0470.0410.059ACC0.8750.8770.8850.8760.8840.8800.8930.8900.903LC2017Q1AUC0.8230.8010.8040.8160.8590.8740.8720.8670.879HM0.3860.2600.3320.2570.3150.4400.3780.4350.350BS0.1580.2370.2520.1950.1820.1570.1960.1490.203ACC0.8610.8880.8770.8750.8710.8780.8710.8730.912From Tables7 and 5, we can see that, compared with the single classifiers, ensemble methods reveal significant advantages with regard to the accuracy of classification. Compared with other single classifiers and combined approaches in both groups, EGHE has an advantage in all metrics across all datasets. Experiments on several state-of-the-art ensemble models are performed to verify the effectiveness of the EGHE model. They are an EMPNGA-based multistage hybrid model put forward by Zhang and Xia [37]; the heterogeneous ensemble credit model put forward by Xia et al. [40]; EBCA-RF&XGB-PSO model that is put forward by He et al. [41]; heterogeneous ensemble learning-based two-stage credit risk model (TSHE) proposed by Papouskova and Hajek [42]; twin neural networks (TNN) proposed by Jayadeva et al. [43]; and a new rule-based knowledge extraction (RKE) method proposed by Mahani and Baba [44] recently. Table 8 gives the results of ensemble models in different data sets.Table 8
Comparison results of ensemble models in different data sets.
Data setPerformance measurementEnsemble modelsEMPNGA-based modelHeterogeneous ensembleEBCA-RF& XGB-PSOTSHETNNRKEEGHEGermanyAUC0.8020.7950.7980.7750.8110.7900.823HM0.4000.3860.3970.3760.3200.3760.325BS0.1580.1640.1630.1840.1810.1580.184ACC0.7680.8590.8690.8390.8730.8740.886AustraliaAUC0.9400.9230.9310.9330.9310.9290.945HM0.6720.6480.6610.6280.6490.6760.659BS0.0920.1010.0920.1010.0950.0990.096ACC0.8750.8420.8610.8520.8820.8770.895JapanAUC0.9320.9250.9190.9350.9230.9210.937HM0.6650.6510.6360.6490.6500.6490.660BS0.0950.0910.0970.0900.0970.0900.098ACC0.8720.8830.8890.8870.8900.8870.904IranAUC0.8760.8240.8310.8240.7900.8210.802HM0.4240.3840.4890.3840.2980.4160.303BS0.0580.0470.0610.0430.0580.0530.059ACC0.9070.9080.9210.9020.9110.9100.915Bene 1AUC0.8240.8210.8180.8210.8680.8190.881HM0.3760.3830.4420.4230.3460.3840.351BS0.1520.1470.1470.1500.1690.1390.173ACC0.8720.8690.8690.8650.8830.8720.896Bene 2AUC0.8660.8710.8630.8810.8740.8750.887HM0.4790.4880.4650.4980.4980.4900.506BS0.1020.0980.1010.1180.1120.1100.114ACC0.8710.8670.8750.8870.8850.8820.898ShuttleAUC0.9130.8700.8500.8650.9290.8500.943HM0.4420.4050.5000.4030.6480.4310.658BS0.0600.0500.0620.0450.0950.0550.096ACC0.8860.8780.8920.8860.9020.8920.916Skin_segmentAUC0.8590.8670.8370.8620.9210.8480.935HM0.3920.4040.4520.4440.6490.3970.659BS0.1580.1550.1500.1580.0960.1430.098ACC0.8700.8770.8890.8980.8940.8930.908MiniBooNEAUC0.9030.9190.8830.9250.7880.9060.800HM0.4990.5150.4760.5230.2970.5070.302BS0.1060.1030.1030.1240.0580.1140.059ACC0.8900.8950.9010.8970.8890.9010.903LC2017Q1AUC0.8580.8460.8220.9230.8660.8450.879HM0.4020.2740.3400.3680.3450.2660.350BS0.1650.2500.2580.2130.1990.2020.203ACC0.8880.8970.9030.8950.8980.9030.912From Table8, we could tell that the results of these models are very close. The accuracy of EGHE model is better than most of the other models but Iranian. The EBCA-RF&XGB-PSO model achieved a high accuracy of 0.921 in the Iranian data set because it uses the Extended Balance Cascade method that can effectively solve the issue of class imbalance. However, the ensemble strategy and GFSS theory-based model EGHE can deal with the thorny problem of unbalanced data classification better in most experimental data sets; even in some severely skewed data sets, ideal outcomes have been achieved, such as in Shuttle, Skin_segment, MiniBooNE, and LC2017Q1.


## 3.1. Preparation of Dataset

During the evaluation, we collected many different private and public data sets. We have collected a total of six credit data sets and four additional imbalanced data sets with different IR are obtained, i.e., three public and three private. The public sets can be obtained from the UCI Machine Learning Repository. They are real-world credit score data sets and are now widely used by researchers. The German, Australian, and Japanese data sets are used for extra verification. The private data sets consist of the Iranian data set that has also been widely used in many studies and the Bene 1 and 2 data sets, which can be obtained from two key financial institutions in Benelux et al. [32]. This Iranian set has various customer data of many small Iranian private banks [33, 34]. Four additional imbalanced datasets are also from the Machine Learning Repository, UCI. They are Shuttle, Skin_segment, MiniBooNE, and LC2017Q1 which contains loan data of the first quarter in 2017 from Lending Club. The characteristics of all experimental data sets can be found in Table 2.Table 2
Characteristics of data sets.
Data setsSizeAttributesGood/badImbalance ratioNumber of classifiersCredit scoring data setsGermany100025700/3002.333Australia69014307/3830.801Japan69015296/3570.831Iran100027950/501919Bene 13,123332,082/1,04122Bene 27,190335,033/2,1572.333Additional imbalanced data setsShuttle12,380911,428/95212.00413Skin_segment117,7283114,039/3,67930.9931MiniBooNE201,35510196,555/4,80040.9541LC2017Q195,6337294,414/1,21977.4578In this paper, we compared our proposed model with the other four state-of-the-art models, namely, C5.0 decision tree, SVM with Radial Basis Function, kernel SVM-R, Deep Belief Networks (DBN), and Bayes to validate the performance of our approaches. All the continuous attributes will be discretized into various intervals. Every single data set will be divided into a two-thirds training set and one-third testing set randomly. We use the open source platformR-statistics (version R-3.2.2.) to conduct our experiments.


## 3.2. Experimental Results

Different methods are utilized as comparison models to test the validity of the EGHE credit scoring model.Firstly, the FS algorithm that is based on AEnet is used to obtain the highly correlated variables after initial data gathering and preprocessing. We could notice that after selection, the variables in these ten data sets are all decreased to various degrees (Table3). In consideration of the complexity of computation, deleting irrelevant or weakly correlated variables is becoming increasingly important for big-data-oriented credit assessment issues.Table 3
Number of features before and after AEnet-based selection.
Data setsNumber of featuresBeforeAfterGermany2521Australia1410Japan1512Iran2722Bene 13315Bene 23320Shuttle1613Skin_segment33MiniBooNE109LC2017Q17248After feature selection, we can perform two experiments to further see the effect of feature selection on the classification of each model: (1) all single classifiers with feature selection and (2) all single classifiers without feature selection. Tables4 and 5 show in detail the AUC, H-measure (HM), Brier’s score (BS), and accuracy (ACC) for all single classifiers with and without feature selection.Table 4
Results for all single classifiers without AEnet-based feature selection.
Data setPerformance measuresSingle classifiersC5.0SVM-RDBNBayesELMGermanyAUC0.6800.7320.7530.7610.764HM0.3360.3620.3990.3380.379BS0.2510.1670.1730.1990.172ACC0.7450.7520.7490.7450.762AustraliaAUC0.8670.8120.8760.8570.852HM0.5160.6030.6130.5810.610BS0.1480.1130.1070.1660.127ACC0.7360.7420.7510.7340.735JapanAUC0.8570.9080.9110.8880.882HM0.6890.6080.6080.6670.618BS0.1560.1200.1380.1760.142ACC0.7380.7490.7570.7420.751IranAUC0.6160.6040.6140.7150.710HM0.1110.0740.0610.1920.133BS0.0710.0710.0780.0770.083ACC0.6230.6170.6400.6270.648Bene 1AUC0.7280.8210.7670.7400.752HM0.3100.3660.2500.3350.351BS0.2670.1760.1680.2970.199ACC0.7010.7490.6980.6900.753Bene 2AUC0.7450.7890.7460.7070.762HM0.3850.3090.2380.2760.270BS0.1430.1780.1420.1720.153ACC0.7280.7200.7150.7110.747ShuttleAUC0.6040.6750.6930.7140.722HM0.2990.3340.3670.3170.358BS0.2240.1530.1580.1860.163ACC0.6620.6930.7400.6980.720Skin_segmentAUC0.7700.7490.8040.8020.805HM0.4580.5560.5640.5430.576BS0.1310.1050.0990.1560.119ACC0.6530.6850.6890.6870.695MiniBooNEAUC0.7610.8370.8380.8330.833HM0.6130.5600.5590.6250.584BS0.1400.1120.1260.1640.134ACC0.6550.6900.6960.6950.710LC2017Q1AUC0.5470.5570.5630.6690.671HM0.0990.0670.0570.1810.126BS0.0620.0660.0720.0710.078ACC0.6550.6700.6880.6870.712Table 5
Results for all single classifiers with AEnet-based feature selection.
Data setPerformance measurementSingle classifiersC5.0SVM-RDBNBayesELMGermanyAUC0.6900.7560.7620.7740.768HM0.1810.2950.2480.2670.268BS0.2300.1630.1720.1920.173ACC0.7510.7590.7520.7580.765AustraliaAUC0.8820.9020.9150.9120.883HM0.6140.6350.6420.6230.624BS0.1210.1040.1310.1210.130ACC0.7490.7500.7550.7410.748JapanAUC0.8610.9100.9190.9100.878HM0.6080.6210.6310.6220.622BS0.1160.1120.1260.1130.119ACC0.7490.7520.7660.7530.758IranAUC0.6310.6500.6120.7230.638HM0.1370.1180.1080.2120.117BS0.0680.0710.0780.0720.081ACC0.6520.6450.6500.6520.658Bene 1AUC0.7700.8100.8060.7730.798HM0.3120.3450.3360.3020.320BS0.2330.1770.1840.2640.241ACC0.7210.7460.7190.7210.753Bene 2AUC0.7630.8240.8150.7800.796HM0.3540.3860.2690.2850.326BS0.1320.1190.1370.1250.137ACC0.7340.7340.7150.7230.747ShuttleAUC0.6520.6820.6860.7230.729HM0.3230.3370.3630.3210.362BS0.2420.1550.1560.1880.165ACC0.6850.6990.7430.7070.727Skin_segmentAUC0.8320.7560.7960.8120.813HM0.4950.5620.5580.5500.582BS0.1410.1060.0980.1580.120ACC0.6750.6920.7120.6960.702MiniBooNEAUC0.8220.8450.8300.8440.841HM0.6620.5660.5530.6330.589BS0.1510.1130.1250.1660.135ACC0.7070.7040.6890.7040.717LC2017Q1AUC0.5910.5630.5570.6780.678HM0.1070.0680.0560.1830.125BS0.0670.0670.0710.0720.079ACC0.6820.6870.6810.6960.719From the results of every single classifier in Tables4 and 5, we can see that the feature selection helps enhance the effectiveness of classification of most single classifiers. After feature selection, C5.0 shows accuracy values that are 0.8%, 1.76%, 1.63%, 4.49%, 2.85%, 0.82%, 3.47%, 4.15%, 7.94%, and 4.12% greater than without feature selection for Germany, Australia, Japan, Iran, Bene 1, Bene 2, Shuttle, Skin_segment, MiniBooNE, and LC2017Q1 data sets, respectively. In the same way, SVM-R increased the accuracy values by 0.93%, 1.08%, 0.67%, 4.53%, −0.4%, 1.94%, 0.87%, 1.02%, 2.03%, and 2.54%, respectively, for ten experimental data sets. DBN and Naïve Bayes improve their effectiveness too after feature selection. Only SVM-R on Bene 1 reduced by 0.4%, but this does not contradict the improvement on the classification that our AEnet-based feature selection brings. Not only do outliers, redundant, and weakly or even unrelated variables help improve the effectiveness, but also affect the model establishment and cause great computation.It is noteworthy that, compared with C5.0, SVM-R, DBN, and Bayes, ELM has manifested the superiority in accuracy on the vast majority of data sets. Table6 reports the average running time (total time for training and testing) for all models. For these experiments, we use an Intel i5-8500 with CPU at 3.0 GHz and 16 GB of RAM.Table 6
Running time for all models with feature selection (seconds).
Data setsModelsC5.0SVM-RDBNBayesELMGermany1.762.894.331.721.38Australia1.162.022.991.070.97Japan1.252.052.901.130.94Iran1.823.014.311.691.44Bene 15.6310.1114.295.854.62Bene 212.7121.1328.9612.3210.01Shuttle21.7935.7853.6121.2917.11Skin_segment207.20340.24509.76202.49182.46MiniBooNE354.38581.92871.87346.33277.87LC2017Q1591.26970.781254.49560.57436.04From Table6, we can see that ELM costs less time than other single models to carry out credit scoring activities. The efficiency of computing resources also makes ELM a great match for ensemble learning and modeling.After implementing feature selection, completing the ensemble strategy, and individual model classification, the EGHE model can be achieved. Based on (14), (15), and (17)–(19), the weights of single ELM classifiers are calculated according to their efficiency, respectively. To validate the availability of EGHE, we employed several ensemble models in contrast with EGHE. These models were split into two parts. The first one contains four FS algorithms with GFSS-based combination. They are cost-sensitive, GA, information gain ratio (IGR), and elastic net (Enet). Cost-sensitive, GA, and IGR are popular feature selection approaches in the credit scoring area [16, 35, 36]. The second group applies four other approaches with AEnet-based feature selection, which were weighted average (WAVG) [37], majority voting (MajVot) [38], weighted voting (WVOT) [39], and fuzzy soft set (FSS). Those methods are frequently adopted in the establishment and utilization of different combination models. They also employ ELM as the classifier but did not take the ensemble strategy that is proposed above, only using random sampling methods to make all training data sets become balanced. Table 7 displays the results of AUC, H-measure, and Brier’s score for all ensemble models.Table 7
Classification outcomes from all data sets for different performance measures with various combinations.
Data setPerformance measurementOther feature selection methods with GFSS-based combinationTraditional combination methods with AEnet-based feature selectionEGHECost-sensitiveGAIGREnetWAVGMajVotWVOTFSSGermanyAUC0.7860.7920.7770.7810.7770.7860.7890.7710.823HM0.2860.1770.2260.2380.2220.2850.2460.3050.325BS0.1660.2080.1970.1820.1810.1830.1920.1570.184ACC0.8420.8620.8330.8270.8450.8590.8360.8440.886AustraliaAUC0.9210.9270.9330.9280.9210.9220.9320.9280.945HM0.6670.5190.5730.6280.6370.6370.6320.6530.659BS0.0920.1450.1730.1030.1010.1120.1070.0970.095ACC0.8720.8780.8820.8630.8720.8730.8760.8740.895JapanAUC0.9230.9280.9180.9320.9180.9270.9280.9250.937HM0.6500.4920.5660.6180.6020.6420.6170.6480.660BS0.0990.1570.1740.1090.1210.1120.1160.1030.098ACC0.8710.8620.8610.8680.8530.8640.8730.8680.904IranAUC0.7910.7780.7810.7670.7770.7790.7830.7780.802HM0.2790.1500.2170.1620.2830.1080.1070.2940.303BS0.0420.0690.0560.0490.0440.0480.0490.0440.059ACC0.8830.8810.8740.8610.8670.8850.8770.8830.915Bene 1AUC0.8430.8120.8220.8210.8820.8790.8860.8880.881HM0.3960.2630.3380.2580.3240.4410.3850.4470.351BS0.1610.2400.2560.1970.1880.1590.1980.1540.173ACC0.8810.8720.8640.8720.8720.8720.8650.8760.896Bene 2AUC0.9210.8380.8580.8680.8780.8440.8760.8830.888HM0.5370.4010.4780.4890.5050.4360.4320.4960.506BS0.0910.1360.1680.1220.1020.1150.1120.1030.114ACC0.8780.8600.8570.8680.8810.8750.8820.8840.898ShuttleAUC0.8960.9140.9140.9220.8970.9190.9170.9060.943HM0.6530.5100.5620.6250.6200.6350.6220.6360.658BS0.0910.1430.1700.1030.0990.1100.1040.0960.095ACC0.8520.8660.8620.8580.8490.8700.8620.8510916Skin_segmentAUC0.9020.9130.8990.9260.8960.9220.9150.9030.936HM0.6370.4860.5550.6150.5860.6380.6070.6310.659BS0.0960.1540.1710.1070.1190.1100.1130.0980.098ACC0.8530.8490.8440.8630.8330.8620.8590.8450.908MiniBooNEAUC0.7730.7670.7640.7630.7570.7750.7700.7580.800HM0.2720.1490.2130.1590.2770.1090.1040.2860.302BS0.0420.0670.0540.0480.04390.0470.0470.0410.059ACC0.8750.8770.8850.8760.8840.8800.8930.8900.903LC2017Q1AUC0.8230.8010.8040.8160.8590.8740.8720.8670.879HM0.3860.2600.3320.2570.3150.4400.3780.4350.350BS0.1580.2370.2520.1950.1820.1570.1960.1490.203ACC0.8610.8880.8770.8750.8710.8780.8710.8730.912From Tables7 and 5, we can see that, compared with the single classifiers, ensemble methods reveal significant advantages with regard to the accuracy of classification. Compared with other single classifiers and combined approaches in both groups, EGHE has an advantage in all metrics across all datasets. Experiments on several state-of-the-art ensemble models are performed to verify the effectiveness of the EGHE model. They are an EMPNGA-based multistage hybrid model put forward by Zhang and Xia [37]; the heterogeneous ensemble credit model put forward by Xia et al. [40]; EBCA-RF&XGB-PSO model that is put forward by He et al. [41]; heterogeneous ensemble learning-based two-stage credit risk model (TSHE) proposed by Papouskova and Hajek [42]; twin neural networks (TNN) proposed by Jayadeva et al. [43]; and a new rule-based knowledge extraction (RKE) method proposed by Mahani and Baba [44] recently. Table 8 gives the results of ensemble models in different data sets.Table 8
Comparison results of ensemble models in different data sets.
Data setPerformance measurementEnsemble modelsEMPNGA-based modelHeterogeneous ensembleEBCA-RF& XGB-PSOTSHETNNRKEEGHEGermanyAUC0.8020.7950.7980.7750.8110.7900.823HM0.4000.3860.3970.3760.3200.3760.325BS0.1580.1640.1630.1840.1810.1580.184ACC0.7680.8590.8690.8390.8730.8740.886AustraliaAUC0.9400.9230.9310.9330.9310.9290.945HM0.6720.6480.6610.6280.6490.6760.659BS0.0920.1010.0920.1010.0950.0990.096ACC0.8750.8420.8610.8520.8820.8770.895JapanAUC0.9320.9250.9190.9350.9230.9210.937HM0.6650.6510.6360.6490.6500.6490.660BS0.0950.0910.0970.0900.0970.0900.098ACC0.8720.8830.8890.8870.8900.8870.904IranAUC0.8760.8240.8310.8240.7900.8210.802HM0.4240.3840.4890.3840.2980.4160.303BS0.0580.0470.0610.0430.0580.0530.059ACC0.9070.9080.9210.9020.9110.9100.915Bene 1AUC0.8240.8210.8180.8210.8680.8190.881HM0.3760.3830.4420.4230.3460.3840.351BS0.1520.1470.1470.1500.1690.1390.173ACC0.8720.8690.8690.8650.8830.8720.896Bene 2AUC0.8660.8710.8630.8810.8740.8750.887HM0.4790.4880.4650.4980.4980.4900.506BS0.1020.0980.1010.1180.1120.1100.114ACC0.8710.8670.8750.8870.8850.8820.898ShuttleAUC0.9130.8700.8500.8650.9290.8500.943HM0.4420.4050.5000.4030.6480.4310.658BS0.0600.0500.0620.0450.0950.0550.096ACC0.8860.8780.8920.8860.9020.8920.916Skin_segmentAUC0.8590.8670.8370.8620.9210.8480.935HM0.3920.4040.4520.4440.6490.3970.659BS0.1580.1550.1500.1580.0960.1430.098ACC0.8700.8770.8890.8980.8940.8930.908MiniBooNEAUC0.9030.9190.8830.9250.7880.9060.800HM0.4990.5150.4760.5230.2970.5070.302BS0.1060.1030.1030.1240.0580.1140.059ACC0.8900.8950.9010.8970.8890.9010.903LC2017Q1AUC0.8580.8460.8220.9230.8660.8450.879HM0.4020.2740.3400.3680.3450.2660.350BS0.1650.2500.2580.2130.1990.2020.203ACC0.8880.8970.9030.8950.8980.9030.912From Table8, we could tell that the results of these models are very close. The accuracy of EGHE model is better than most of the other models but Iranian. The EBCA-RF&XGB-PSO model achieved a high accuracy of 0.921 in the Iranian data set because it uses the Extended Balance Cascade method that can effectively solve the issue of class imbalance. However, the ensemble strategy and GFSS theory-based model EGHE can deal with the thorny problem of unbalanced data classification better in most experimental data sets; even in some severely skewed data sets, ideal outcomes have been achieved, such as in Shuttle, Skin_segment, MiniBooNE, and LC2017Q1.


## 4. Conclusion

In this paper, we proposed a novel ensemble credit scoring model called EGHE, which integrates efficient feature selection algorithm, novel ensemble strategy, and GFSS-based weighting method for single ELM classifiers. In the proposed model, the adaptive elastic net-based feature selection algorithm was firstly utilized to obtain high-quality training data to improve the evaluation efficiency without reducing the predictive precision. ELM model was employed as basic classifier, and a novel ensemble strategy was generated to make the imbalanced training data sets become balanced for each ELM classifier. Additionally, we proposed a new weighting method to build the GFSS theory-based ensemble credit scoring model. Dual-scale classification accuracy metric that is based on new similarity measurement of GFSS was constructed to compute the final weight of every single classifier. The biggest contribution of this paper is that the proposed EGHE is able to predict credit risk reliably and accurately, especially for unbalanced credit data. Comparisons between EGHE and other credit scoring models were implemented on ten real-world datasets with four metrics (average accuracy, AUC, H-measure, and Brier’s score). A variety of state-of-the-art ensemble models were employed to compare with EGHE to prove its validity. The experiments results demonstrated that the proposed EGHE model was robust and represented a positive development in credit scoring.



---

*Source: 7504764-2020-06-30.xml*