# Novel Key Indicators Selection Method of Financial Fraud Prediction Model Based on Machine Learning Hybrid Mode

Authors: Hongsheng Xu; Ganglong Fan; Yanping Song
Journal: Mobile Information Systems
 (2022)

---

## Abstract

In recent years, the financial fraud event of listed companies has occurred continuously. The financial fraud has brought huge losses to the capital market and investors, hindering the investment allocation mechanism of the capital market. The current financial fraud prediction model can judge the company that may conduct financial fraud in advance. So, this can reduce economic losses. The key factor to construct the financial fraud prediction model is how to select the evaluation indicators. This paper analyzes the existing indicators selection method and finds the problem of low prediction accuracy. A key indicators selection method of prediction model based on machine learning hybrid mode is proposed. First, the contribution degree of the selected algorithm and model is ranked according to the features. The support vector machine with good classification effect and heterogeneity with other models is used as the intermediate evaluation model. A variety of selected indicators from machine learning are tested for AUC on the intermediate model. Well-performing machine learning models are selected and combined into multiple hybrid modes. These hybrid models are tested for AUC again. Experiments demonstrate that the hybrid mode of Lasso method and random forest performed best in the AUC test. The repetition indicators of the hybrid model are then selected as important indicators of the prediction model. Finally, the correlation of the indicators is tested, and the indicators beyond the threshold are removed. The selected key indicators effectively improve the accuracy of financial fraud prediction.


---

## Body

## 1. Introduction

With the high development of social economy, people need more accurate data and information when making economic decisions. The financial indicators provided by the financial report of listed companies are the basis to measure the development status of the enterprise. If more listed companies issued a false financial report. On the one hand, it will seriously mislead investors, affect investors’ investment decisions, and reduce their investment interest. On the other hand, it will greatly reduce the spontaneous resource allocation efficiency of the capital market, harm the development of the whole capital market, and even endanger the benign development of social economy and national economy.Therefore, the financial data of listed companies are analyzed, and the financial fraud prediction model with high accuracy is constructed. It is very necessary to effectively predict the fraud behavior of listed companies. The financial fraud prediction of listed companies can help to truly and accurately reflect the operating conditions of the enterprise, improve the efficiency of the capital market, and promote the healthy and stable development of the capital market.Current research on financial fraud prediction models has been mostly reported. The author compares the application effect of the traditional BP network, decision tree, and other models in the financial statement fraud identification model. By comparing the prediction accuracy of the model, it is concluded that the Bayesian network has a higher prediction accuracy [1]. This paper is a combination of machine learning algorithms to first select fake feature value through artificial neural networks and support vector machines, and then construct financial fraud identification models using four types of decision trees [2]. Nurul Herawati uses M-Score models in the field of financial and financial analytics, combined with data mining technology that can more effectively identify financial fraud behavior [3]. The paper uses the text mining method to analyze the content in the financial report, and finds the fraud information in language structure of the text content [4]. The authors used decision tree and classified regression tree of machine learning models to predict the financial fraud of listed companies in the United States [5]. The experimental results show that the methods such as random forest, support vector machine, and neural network are well used in the financial fraud identification model of listed companies, among which the random forest performs best in the test set [6].The authors firstly used the Mann–Whitney test and correlation analysis with principal component analysis to identify the financial indicators for building the model [7]. The author conducted a financial fraud study on listed companies in Malaysia, involving a total of 65 fraudulent samples and 65 non-fraudulent samples. The purpose of the research is to find suitable financial indicators for predicting financial fraud [8]. This paper builds different financial fraud identification models based on data mining technology, and conducts financial fraud identification tests combined with samples. The experimental results show that the recognition efficiency of indicators combination model of random forest and Relief algorithm is the highest [9]. The author uses a variety of statistical methods to extract features, adopts the method of three neural network fusion models, and uses the AUC value as the evaluation indicators to predict the listed companies with financial fraud [10].In short, the key factor to build a financial fraud prediction model is how to select the evaluation indicators. Then, the existing methods have the problem of low prediction accuracy in the indicators selection. This paper proposes a novel key indicators selection method of financial fraud prediction model based on machine learning hybrid mode. First, the contribution degree of the selected algorithm and model is ranked according to the features. We select these models as Pearson’s coefficient, Lasso method, multiple linear regression model, random forest model, XGBoost, and decision tree. The support vector machine with few parameters and good classification effect is used as the intermediate evaluation model. A variety of selected indicators from machine learning are tested for AUC on the intermediate model. Well-performing machine learning models are selected according to the experimental results and combined into multiple hybrid modes. These hybrid models are tested for AUC again. The optimal hybrid mode is finally determined, and the repetition indicators in this hybrid mode are used as an important indicator of the prediction model. After correlation test of indicators, the indicators beyond the threshold are removed, and the key indicators of financial fraud prediction model are finally obtained. The selected key indicators are more accurate when making fraud predictions.


## 2. Data Preprocessing

The dataset collected in this article comes from 11,310 financial data from 2,660 listed manufacturing companies in the first five years. To prevent the data from involving sensitive content, data masking has been conducted. There are 91 fraud samples in these financial data, and the proportion of samples for whether financial fraud is 1 : 124, similar to the actual situation, the relevant indicators are 361. Finally, use the FLAG column to indicate whether the fraud (0: normal, 1: fake). Here, the collected data is first to read by using Excel to obtain multiple data tables. These tables are merged to remove duplicate attributes, and then deletion of moderately irrelevant features is performed to prevent excessive features from causing overfitting, as well as to prevent noise problems. The specific operation is as follows:(1)
It is considered that the data for each year are relatively independent, regardless of the time series problem, so the three features of “ACT_PUBTIME,” “END_DATE_REP,” and “PUBLISH_DATE” can be deleted(2)
Financial fraud prediction mainly considers financial data, and removes the six features of non-financial indicators “REPORT_TYPE,” “FISCAL_PERIOD,” “ACCOUTING_STANDARDS,” “CURRENCY_CD,” “FISCAL_PERIOD,” and “MERGED_FLAG.” Additional features with a variance of 0 are removed using variance filtering. Finally, we get the financial data sample table, as is shown in Table1Table 1
Sample table of financial data.
HinNOTES_PAYABLEINT_PAYABLEINTAN_ASSETST_LIABOTH_RECEIVCIP…FLAG20260864.1943137262.8935853006.19199014246.63584940.663918166.78…0253001641824324198151920.9374581368285626756.0547840630.38…0150309070.233933660.12161767971.2991771386.237519236.14498490316.2…0135439801.1444138.21133556485.4113231694967249473.03589941.46…037133.337648295.11103051049.6775584.935852628.26…081818076.32122502787.311574909.723776020.83…015867887.418069593.1101698231.22214365.14…04811000052833.3423280391.4179528298.53127042.06408878.15…0191162947.481162818.28762841837.120761013.4930169688.61…082165039.57197499986.412115563.06506346335.6214222579726651977.7112975382.2…057160740.76244048187191111920118937852.67183144656.5…0400000086010001011491004145001079500…0455832602.9357750000433867311.47485796120135391987.6108686448…063346265.5430744297.613692803224100364.7579193425.27…025534371.17330467802.8192516535.62255491128183031871.6134753825.4…04170589061048005.186014023828171552625.7567848801.8…012578448855000000736216.35228561097149590869942011631.461062885591…024267329.857777.7734716946.33132601183.6873642.4933681.85…04147201.25171770425778.788582593.46553913707.143086003.84117853510.9…034324301.94274456.67101165025.8729939458.810358632.6969771256.84…0


### 2.1. Processing of Missing Values

Missing data values is one of the problems frequently encountered in data analysis. For the missing values in Table1, models and algorithms may not be used for the data without processing. At the same time, improper methods and means of processing missing values may lose a lot of information, which may get wrong conclusions in data analysis. Missing data is divided into three categories: MCAR (Missing Completely at Random), MAR (Missing at Random), and MNAR (Missing not at Random). How to fill in the missing values is a key issue in the research.


#### 2.1.1. Two Methods of Filling in the Missing Values

(1) Delete the Missing Value. There are mainly simple deletion methods and weight methods. The simple deletion method is to directly delete samples with missing values, which is the most direct way to delete data. This method applies to a large sample size but a small missing proportion (such as 5%); the weighting method means that when the type of the missing values is MNAR, bias can be reduced by weighting the complete data. After marking the incomplete data, the complete data cases are given different weights, and the weight of the cases can be obtained by logistic or probit regression. The weighting method is not ideal for multiple missing properties.(2) Possible Value Interpolates the Missing Values. The main idea is to interpolate missing values with the most likely values than less information loss resulting from full deletion of incomplete samples. There are several common methods:
(1)
Mean interpolation. The properties of the data are divided into numerical and non-numerical methods. If the missing value is numerical type, the missing value is interpolated with the average of the values for that feature; if the missing value is non-numerical type, the missing value is supplemented with the mode (the value with the highest frequency)(2)
Mean interpolation of the same kind. The use of mean interpolation has a disadvantage: all missing values on attributext containing missing values are filled with the mean value of the attribute, which may lead to a decrease in accuracy when the classification algorithm is performed subsequently. The idea of similar mean interpolation is to first classify the samples and then interpolate the missing values with the mean of the samples of that classKnown datasetD=x1,y1,x2,y2⋯xN,yN, where xi=xi1,xi2,⋯xidT,i=1,2,⋯,N. Assuming attribute xt contains missing values, divides the dataset into Dl=x1,y1,x2,y2⋯xl,yl and Du=x1,y1,x2,y2⋯xu,yu, where xt contains valid values on Dl and missing values on Du.First, use the hierarchical clustering algorithm to clusterDl. Let the result of the clustering be k clusters, calculate the mean value μ1,μ2⋯,μk of these k clusters on xt.Forxi∈Dl, xi∧t=xit.Forxi∈Du, first cluster it, assuming it belongs to cluster Ck1≤k≤K, then xi∧t=μk.
(3)
ML (Max Likelihood). Under conditions with random missing, assuming that the model is correct for the complete sample, a maximum likelihood estimation of the unknown parameters can be performed by the marginal distribution of the observed data. The usual calculation method of parameter estimation for maximum likelihood is EM (Expectation-Maximization). The proposed method is suitable for large samples. But this approach can fall into local extrema, convergence is not fast, and computationally complex


#### 2.1.2. Processing of the Missing Values in This Paper

Due to the uneven miss rate of the rows and columns of the dataset, the number of fake samples and normal samples is greatly different. What’s more, the causes of the Loss rate of Row are different from the Loss rate of Columns, so we used different methods for rows and columns.(1)
For column data missing are shown in Figure1. This paper believes that the absence of serious financial feature indicates that this feature plays little role in judging financial fraud in the financial analysis, so removes the indicators with a missing rate of greater than 70%(2)
For the row loss rate as shown in Figure2. The mean value of the sample attribute of whether it is fake or not may be quite different. And the classification algorithm is used in this paper. In order to prevent the blurring of the discriminant boundary in the filling of missing values, we use a similar mean to interpolate the missing values, and when the loss rate is too large, using the similar mean interpolation causes the excessive error. The practice of this article is the samples are first divided into two groups of normal data and fake data group according to whether the sample is fake, the mean filling is used for samples with normal data missing rates within 70%; samples with a missing rate of more than 70% are deleted; for the absence of fake data, because of the few data being fake, the deletion will lead to the poor generalization ability of the construction model, so we keep all the fake samples, and the missing value is filled with the meanFigure 1
Scatter plot of the column missing rate.Figure 2
Scatter plot of the row missing rate.


### 2.2. Data Standardization

The samples selected in this paper are mainly numerical indicators, including different categories of numerical indicators, and there are large differences between different factors in the range of values and units of measure so that we are unable to compare, weight, and other subsequent operations on the different indicators, so we need to standardize the indicators data.Common data standardization methods are Min-Max, Z-score, decimal scaling, quantitative feature binarization, etc. Since the data selected in this paper may have extremes, we use the Z-score normalization method. Z-score normalization is a normalization method to transform the data into standard normal distribution. The specific calculation formula is:(1)x¯=x−x¯stdx,where thex¯ represents the mean value and stdx represents the standard deviation.Considering that the following situation may occur in the real environment: ① data will be continuously input into the model, and the mean and variance cannot be obtained; ② the training set is simulating data in a real environment and cannot directly use, it is own mean and variance; ③ in real environments, single data cannot be normalized. To solve these three problems, we first obtain the parameters (mean, variance) in the training set; the entire dataset is then standardized using the Z-score normalization method. The specific flow is shown in Figure3.Figure 3
Standardized flow figure of sample data.In Figure3, fit represents the mean and variance obtained according to the training dataset, returning a Scalar object; Transform means that according to the obtained mean and variance, the Z-score method is used to standardize the training set and test set data at the same time. The final result is obtained.Finally, the dataset treated with missing values is normalized using the above Z-score normalization method in the experiment, and the data results are shown in Table2.Table 2
Financial data schedule.
QuantityFake samples8598Normal samples91Factors106


## 2.1. Processing of Missing Values

Missing data values is one of the problems frequently encountered in data analysis. For the missing values in Table1, models and algorithms may not be used for the data without processing. At the same time, improper methods and means of processing missing values may lose a lot of information, which may get wrong conclusions in data analysis. Missing data is divided into three categories: MCAR (Missing Completely at Random), MAR (Missing at Random), and MNAR (Missing not at Random). How to fill in the missing values is a key issue in the research.


### 2.1.1. Two Methods of Filling in the Missing Values

(1) Delete the Missing Value. There are mainly simple deletion methods and weight methods. The simple deletion method is to directly delete samples with missing values, which is the most direct way to delete data. This method applies to a large sample size but a small missing proportion (such as 5%); the weighting method means that when the type of the missing values is MNAR, bias can be reduced by weighting the complete data. After marking the incomplete data, the complete data cases are given different weights, and the weight of the cases can be obtained by logistic or probit regression. The weighting method is not ideal for multiple missing properties.(2) Possible Value Interpolates the Missing Values. The main idea is to interpolate missing values with the most likely values than less information loss resulting from full deletion of incomplete samples. There are several common methods:
(1)
Mean interpolation. The properties of the data are divided into numerical and non-numerical methods. If the missing value is numerical type, the missing value is interpolated with the average of the values for that feature; if the missing value is non-numerical type, the missing value is supplemented with the mode (the value with the highest frequency)(2)
Mean interpolation of the same kind. The use of mean interpolation has a disadvantage: all missing values on attributext containing missing values are filled with the mean value of the attribute, which may lead to a decrease in accuracy when the classification algorithm is performed subsequently. The idea of similar mean interpolation is to first classify the samples and then interpolate the missing values with the mean of the samples of that classKnown datasetD=x1,y1,x2,y2⋯xN,yN, where xi=xi1,xi2,⋯xidT,i=1,2,⋯,N. Assuming attribute xt contains missing values, divides the dataset into Dl=x1,y1,x2,y2⋯xl,yl and Du=x1,y1,x2,y2⋯xu,yu, where xt contains valid values on Dl and missing values on Du.First, use the hierarchical clustering algorithm to clusterDl. Let the result of the clustering be k clusters, calculate the mean value μ1,μ2⋯,μk of these k clusters on xt.Forxi∈Dl, xi∧t=xit.Forxi∈Du, first cluster it, assuming it belongs to cluster Ck1≤k≤K, then xi∧t=μk.
(3)
ML (Max Likelihood). Under conditions with random missing, assuming that the model is correct for the complete sample, a maximum likelihood estimation of the unknown parameters can be performed by the marginal distribution of the observed data. The usual calculation method of parameter estimation for maximum likelihood is EM (Expectation-Maximization). The proposed method is suitable for large samples. But this approach can fall into local extrema, convergence is not fast, and computationally complex


### 2.1.2. Processing of the Missing Values in This Paper

Due to the uneven miss rate of the rows and columns of the dataset, the number of fake samples and normal samples is greatly different. What’s more, the causes of the Loss rate of Row are different from the Loss rate of Columns, so we used different methods for rows and columns.(1)
For column data missing are shown in Figure1. This paper believes that the absence of serious financial feature indicates that this feature plays little role in judging financial fraud in the financial analysis, so removes the indicators with a missing rate of greater than 70%(2)
For the row loss rate as shown in Figure2. The mean value of the sample attribute of whether it is fake or not may be quite different. And the classification algorithm is used in this paper. In order to prevent the blurring of the discriminant boundary in the filling of missing values, we use a similar mean to interpolate the missing values, and when the loss rate is too large, using the similar mean interpolation causes the excessive error. The practice of this article is the samples are first divided into two groups of normal data and fake data group according to whether the sample is fake, the mean filling is used for samples with normal data missing rates within 70%; samples with a missing rate of more than 70% are deleted; for the absence of fake data, because of the few data being fake, the deletion will lead to the poor generalization ability of the construction model, so we keep all the fake samples, and the missing value is filled with the meanFigure 1
Scatter plot of the column missing rate.Figure 2
Scatter plot of the row missing rate.


## 2.1.1. Two Methods of Filling in the Missing Values

(1) Delete the Missing Value. There are mainly simple deletion methods and weight methods. The simple deletion method is to directly delete samples with missing values, which is the most direct way to delete data. This method applies to a large sample size but a small missing proportion (such as 5%); the weighting method means that when the type of the missing values is MNAR, bias can be reduced by weighting the complete data. After marking the incomplete data, the complete data cases are given different weights, and the weight of the cases can be obtained by logistic or probit regression. The weighting method is not ideal for multiple missing properties.(2) Possible Value Interpolates the Missing Values. The main idea is to interpolate missing values with the most likely values than less information loss resulting from full deletion of incomplete samples. There are several common methods:
(1)
Mean interpolation. The properties of the data are divided into numerical and non-numerical methods. If the missing value is numerical type, the missing value is interpolated with the average of the values for that feature; if the missing value is non-numerical type, the missing value is supplemented with the mode (the value with the highest frequency)(2)
Mean interpolation of the same kind. The use of mean interpolation has a disadvantage: all missing values on attributext containing missing values are filled with the mean value of the attribute, which may lead to a decrease in accuracy when the classification algorithm is performed subsequently. The idea of similar mean interpolation is to first classify the samples and then interpolate the missing values with the mean of the samples of that classKnown datasetD=x1,y1,x2,y2⋯xN,yN, where xi=xi1,xi2,⋯xidT,i=1,2,⋯,N. Assuming attribute xt contains missing values, divides the dataset into Dl=x1,y1,x2,y2⋯xl,yl and Du=x1,y1,x2,y2⋯xu,yu, where xt contains valid values on Dl and missing values on Du.First, use the hierarchical clustering algorithm to clusterDl. Let the result of the clustering be k clusters, calculate the mean value μ1,μ2⋯,μk of these k clusters on xt.Forxi∈Dl, xi∧t=xit.Forxi∈Du, first cluster it, assuming it belongs to cluster Ck1≤k≤K, then xi∧t=μk.
(3)
ML (Max Likelihood). Under conditions with random missing, assuming that the model is correct for the complete sample, a maximum likelihood estimation of the unknown parameters can be performed by the marginal distribution of the observed data. The usual calculation method of parameter estimation for maximum likelihood is EM (Expectation-Maximization). The proposed method is suitable for large samples. But this approach can fall into local extrema, convergence is not fast, and computationally complex


## 2.1.2. Processing of the Missing Values in This Paper

Due to the uneven miss rate of the rows and columns of the dataset, the number of fake samples and normal samples is greatly different. What’s more, the causes of the Loss rate of Row are different from the Loss rate of Columns, so we used different methods for rows and columns.(1)
For column data missing are shown in Figure1. This paper believes that the absence of serious financial feature indicates that this feature plays little role in judging financial fraud in the financial analysis, so removes the indicators with a missing rate of greater than 70%(2)
For the row loss rate as shown in Figure2. The mean value of the sample attribute of whether it is fake or not may be quite different. And the classification algorithm is used in this paper. In order to prevent the blurring of the discriminant boundary in the filling of missing values, we use a similar mean to interpolate the missing values, and when the loss rate is too large, using the similar mean interpolation causes the excessive error. The practice of this article is the samples are first divided into two groups of normal data and fake data group according to whether the sample is fake, the mean filling is used for samples with normal data missing rates within 70%; samples with a missing rate of more than 70% are deleted; for the absence of fake data, because of the few data being fake, the deletion will lead to the poor generalization ability of the construction model, so we keep all the fake samples, and the missing value is filled with the meanFigure 1
Scatter plot of the column missing rate.Figure 2
Scatter plot of the row missing rate.


## 2.2. Data Standardization

The samples selected in this paper are mainly numerical indicators, including different categories of numerical indicators, and there are large differences between different factors in the range of values and units of measure so that we are unable to compare, weight, and other subsequent operations on the different indicators, so we need to standardize the indicators data.Common data standardization methods are Min-Max, Z-score, decimal scaling, quantitative feature binarization, etc. Since the data selected in this paper may have extremes, we use the Z-score normalization method. Z-score normalization is a normalization method to transform the data into standard normal distribution. The specific calculation formula is:(1)x¯=x−x¯stdx,where thex¯ represents the mean value and stdx represents the standard deviation.Considering that the following situation may occur in the real environment: ① data will be continuously input into the model, and the mean and variance cannot be obtained; ② the training set is simulating data in a real environment and cannot directly use, it is own mean and variance; ③ in real environments, single data cannot be normalized. To solve these three problems, we first obtain the parameters (mean, variance) in the training set; the entire dataset is then standardized using the Z-score normalization method. The specific flow is shown in Figure3.Figure 3
Standardized flow figure of sample data.In Figure3, fit represents the mean and variance obtained according to the training dataset, returning a Scalar object; Transform means that according to the obtained mean and variance, the Z-score method is used to standardize the training set and test set data at the same time. The final result is obtained.Finally, the dataset treated with missing values is normalized using the above Z-score normalization method in the experiment, and the data results are shown in Table2.Table 2
Financial data schedule.
QuantityFake samples8598Normal samples91Factors106


## 3. Feature Selection Methods Based on Multiple Machine Learning Models

The most appropriate feature selection method for the financial fraud prediction model is the feature selection based on multiple machine learning algorithms and models, such as Pearson, LR, RF, and DT. These machine learning models themselves have a mechanism of scoring features, which are easily applied to feature selection tasks. We first introduce the feature selection based on the machine learning model. Finally, these algorithms and models are used to obtain the relevant features.


### 3.1. Feature Selection Method Based on Machine Learning Model

#### 3.1.1. Feature Selection Based on Pearson’s Correlation

Pearson’s correlation coefficient measures the magnitude of a degree of linear correlation, and the greater the absolute value of the correlation coefficient is, the stronger the degree of linear correlation is. The range of the correlation coefficients is found between [-1,1]. Assuming two variables,X and Y, then there are:
(i)
TheX and Y variables are not correlated when the correlation coefficient is 0(ii)
When the values ofX and Y values increase or decrease at the same time, the two variables are positively correlated, and the correlation coefficients are between 0 and 1(iii)
When the value ofX increases and the Y value decreases, or the X value decreases while the Y value increases; the two variables have a negative correlation, with correlation coefficients between-1 and 0Its formula is:(2)ρX⋅Y=∑X‐X¯Y‐Y¯∑X‐X¯2∑Y‐Y¯2.In formula (2), X and Y are two variables, respectively. X¯ and Y¯ denote the mean values of X and Y, respectively.The correlation coefficient is defined only when the standard deviation of both variables is not zero. Scope of application of Pearson’s correlation coefficient:(1)
There is a linear relationship between the two variables, both with continuous data(2)
Overall normal distribution of two variables, or near-normal unimodal distribution(3)
The two variables are in pairs and each pair is independent of each other


#### 3.1.2. Feature Selection Based on Lasso

Lasso (least absolute shrinkage and selection operator) is a regression method suitable for multicollinearity problems and can implement feature selection while parameter estimation. The Lasso method is a compression estimation method of reducing the set of variables. It constructs a penalty function that can compress the coefficients of variables and change some regression coefficients to 0, thus achieving the purpose of variable selection [11].Lasso regression is performed by imposing a penalty term on the coefficients of the model, so that some coefficients tend to 0 based on the least-squares estimation, to achieve the purpose of variable selection, and also avoid overfitting, ensure the interpretability and simplicity of the model.Consider the following linear model:(3)Y=Xβ+ε.Y is the vector of n×1, X is the matrix of n×p, and ε is the vector of n×1.β=β1,β2,⋯,βp is the regressive coefficient variable of p×1. Its Lasso is estimated at:
(4)β^=argminβ∥Y−Xβ∥22+λ∑j=1pβjAmong them,λ∑j=1pβj is the penalty term, and λ>0 is the reconciliation parameter, used to control the penalty strength of the model and in turn to control the number of explained variables. When λ is small, the weight ratio of the first part of the upper formula will increase, to minimize the sum of the overall residual squares, so that more explained variables will be added to the model; when λ is large, the weight ratio of the second part of the formula above increases, the regression coefficients of many explanatory variables are compressed and tend to be 0.


#### 3.1.3. Multiple Linear Regression Feature Selection

The multiple linear regression model is a model that explains the dependent variable by using two or more explanatory variables. LetY is the dependent variable, X1X2⋯,Xk are k different variables described as explanatory variables. Where X1 is constantly equal to 1, the multiple linear regression model is performed as in Equation (5):
(5)Yi=β1+β2X2i+⋯+βkXki+μii=1,2,⋯,n,where μii=1,2,⋯,n is a random perturbation term; the parameter β1,β2,⋯,βk is called a regression coefficient. Hypothesis,Y=Y1Y2⋮Yn,X=X11X21⋯Xk1X12X22⋯Xk2⋮⋮⋮⋮X1nX2n⋯Xkn,β=β1β2⋮βk,μ=μ1μ2⋮μn. Then, formula (5) is expressed in the matrix form as formula (6).
(6)Y=Xβ+μThe relationship between one or more independent variables is modeled using the least-squares function. Regression analysis in mathematical statistics is a statistical analysis method used to determine the quantitative interdependent relationship between two or more variables. According to the principle of the least-squares method, the estimate ofβi(i=0,1,2,⋯,k), the value of bi(i=0,1,2,⋯,k), should be let by
(7)Q=∑a=1nya−ya∧2=∑a=1nya−b0+b1x1a+b2x2a+⋯+bkxka2⟶min.According to the definition of the least-squares method, in the linear regression model, the estimate of the regression coefficient that minimized the residual sum of squares is called the least-squares-estimation. Equivalent to the smallestβ∧ that makes μ∧′μ∧=Y−Xβ∧′Y−Xβ∧ up, among it, μ′∧ is the transpose of μ∧. To minimize β∧, μ∧′μ∧=Y−Xβ∧′Y−Xβ∧ can be seen as a function about β∧, then the first-order partial derivative of β∧ must be 0, namely, ∂μ∧′μ∧/∂β∧=−2X′Y+2X′Xβ∧=0. Thus, we get equation X′Xβ∧=X′Y, and so β∧=X′X−1X′Y.


#### 3.1.4. Feature Selection Based on Random Forest Model

In machine learning, random forest (RF) is a classifier containing multiple decision trees, and the category of its output is determined by the mode of categories output by the individual trees, and the underlying classifier that constitutes the random forest is called the decision tree. The random forest has the advantages of high accuracy, robustness, and easy to use, making it one of the most popular machine learning algorithms today. There are two ways to calculate feature importance in random forest: one method is based on OOB error, called MDA (Mean Decrease Accuracy); another method is based on Gini impurity, called MDG (Mean Decrease Gini). Both methods are the more the numerical decreases and the more important the representation features [12].The MDA is specifically described below:(1)
A random forest model is trained to test the OOB error for each tree in the model using the out-of-bag sample data(2)
The value of the variable v in the out-of-bag sample data is randomly shuffled to retest the OOB error of each tree(3)
The mean of the difference in OOB error for the two tests is the measure of the importance of a single tree to the variable vThe calculation formula is:(8)MDAv=1ntree∑terrOOBt−errOOBt′.The MDG is specifically described below: Gini-based variable importance is measured by the degree of reduced Gini purity due by use of variablev. At classification node t, the Gini coefficient impurity is:
(9)Gt=1−∑K=1Qp2k/,where Q represents the total number of categories for the target variable, pk/trepresents the conditional probability that the target variable is class k in the node t. The Gini nonpurity drop value for each tree was calculated from the formula, and then the results were averaged across all the trees.


#### 3.1.5. XGBoost Feature Selection

The extreme gradient lifting algorithm (XGBoost) is a tree-based Boosting algorithm [13]. Compared with the traditional gradient improvement decision tree algorithm, the XGBoost algorithm innovatively uses the second derivative information of the loss function, so that the XGBoost algorithm can converge faster, ensure high solution efficiency, and also increase the scalability. Because a function meets the second-order derivable condition, the function can be used as a custom cost function when appropriate. Another advantage of the XGBoost algorithm is that it borrows the column sampling method in the random forest algorithm and further reduces the computation and overfitting.XGBoost calculates which feature to select as a segmentation point based on the gain of the structure fraction, and the importance of a feature is the sum of its occurrences in all trees. That is, the more times an attribute is used to build a decision tree in a model, the relative importance it is. The XGBoost algorithm can be expressed as:(10)yi^=∑k=1kfkxi,fk∈F,where K indicates the number of trees, F=fx=ωqxq:Rm⟶T,ω∈RTrepresents the function space of the model, and fKxi represents the classification results of the i sample in the K tree. As seen from the expression of the XGBoost algorithm, the model is a collection of iterative residual trees, one tree is added to each iteration, and each tree eventually forms the linear combination of the K tree by learning the residuals of the former N−1 tree.


#### 3.1.6. Feature Selection Based on Decision Tree

The structure is similar to that of the tree and consists of oriented edges and nodes. There are three main types of nodes, include the root node, intermediate node, and leaf node. At the top of the decision tree is the root node, which contains the most informative properties; at the bottom of the decision tree are the leaf nodes, representing the results of the classification; between the root and leaf nodes are intermediate nodes used for the testing of feature properties [14].When using the decision tree, the original sample is divided into the training set and the test set, first training a decision tree with the strongest generalization ability, and then predicting using the test set to calculate the generalization error. When training a decision tree, feature selection is about deciding whether to use an indicator as a division basis (as an intermediate node) to help with the classification. The general decision tree algorithm relies on three criteria, namely, information gain, information gain ratio, and Gini index.Information entropy is an indicator used to measure the uncertainty of sample sets, and the more uncertain the set is, the greater the information entropy. Assuming that the proportion of classi samples in sample set D is pii=1,2,3,⋯,y, then the information entropy of sample set D is defined as:
(11)EntD=−∑i=1ypilog2pi.For datasetD, assuming that feature A is selected as the decision tree judgment node, then the information entropy after the action of feature A is defined as:
(12)EntAD=−∑j=1kDjD×EntDj.It was found that decision trees would tend to choose those features with more attribute values when adopting information gain as the criterion for feature selection. To reduce the possible adverse effects of this decision tree preference, the C4.5 decision tree algorithm improves on the original decision tree algorithm. It uses the information gain ratio as a criterion for selecting the optimal partition properties. Information gain ratio is defined as:(13)Gain_rationD,A=GainD,AIVA.Where in equation (13), IVA is known as a fixed value of feature A:
(14)IVA=−∑j=1kDjD×log2DjD.Gini index is also a criterion for a decision tree to select the optimal partition property, representing the probability that a randomly selected sample is misclassification in the set of samples. For sample setD, assuming K categories, the probability of samples belonging to the Kth category is set to A. Then, the Gini index of this probability distribution is:
(15)Ginip=∑k=1kpk1−pk=1−∑k=1kpk2.For the datasetD, its Gini index is:
(16)GiniD=1−∑k=1kCkD2,where Ck represents the number of samples belonging to category k in the sample set D.We have a test at each internal node representation on one attribute by using the tree structure of the decision tree based on the selected features, and test the output on the branches of the tree. Child nodes are generated recursively from top to bottom according to the selected feature evaluation criteria until the decision tree stops growing when the dataset is not separable. The necessary pruning is performed to narrow down the tree structure and alleviate overfitting.


### 3.2. Experiments of Important Feature Selection

The training set and the test set are divided by using the train_test_split method of model_selection in machine learning sklearn; the ratio of the training set to the test set is determined as 7 : 3. To prevent the sample division from affecting the prediction results, random seeds are set when dividing the training set and the test sets. “FLAG” serves as a label column, and other indicators serve as feature columns. Using the data from the training set, for the Pearson correlation coefficient method, all correlation indicators and label columns are imported to find the top 20 features for the correlations with the label column; For the Lasso method, a total of nonzero16 metrics with a weight coefficient are found; the top 20 features are ranked on the model using all correlation metrics to find the top features, respectively. Features are ranked using feature importance on the LR, RF, XGBoost, and DT models, respectively, finding the top 20 features. The main purpose is to obtain the top 20 attributes using sklearn’s ranking of feature_importances_ attribute values. The inherent mechanism of DT is to discriminate the effect of features on the purity increase of nodes based on the actual discriminating criteria such as GINI, information entropy, and information entropy gain. The parameters used on LR, RF, XGBoost, DT models for indicators selection are shown in Table3. These two algorithms of Pearson’s and Lasso do not use parameters. Table 3 shows the names and values of these parameters.Table 3
Parameters list of machine learning model.
NoModel nameParametersNameValue1LRpenaltyL1C1.0solverliblinearmulti_classover2RFn_estimators10max_features20splitterbest3XGBoostboostergbtreemax_depth6learning_rate0.34DTn_estimators10splitterbestmax_depth3Finally, through experiments, the important indicators related to financial fraud are selected by each algorithm and model, as shown in Table4.Table 4
Selection table of important indicators based on machine learning methods.
NumModelPearsonLassoLRRFXGBoostDT1ARINT_PAYABLET_LIAB_EQUITYINT_PAYABLET_ASSETSAVAIL_FOR_SALE_FA2PREPAYMENTT_LIABT_PROFITT_LIABT_LIABT_LIAB3OTH_RECEIVC_OUTF_FR_INVEST_AT_COGSC_OUTF_FR_INVEST_AN_CE_END_BALINT_PAYABLE4AVAIL_FOR_SALE_FAINTAN_ASSETSN_CF_FR_INVEST_AINTAN_ASSETSAVAIL_FOR_SALE_FAN_CE_END_BAL5CIPLT_EQUITY_INVESTN_CF_OPERATE_ALT_EQUITY_INVESTLT_EQUITY_INVESTT_REVENUE6DEFER_TAX_ASSETSDILUTED_EPSN_CF_FR_FINAN_ADILUTED_EPSREVENUEC_OUTF_FR_FINAN_A7OTH_NCANOPERATE_EXPN_INCOMENOPERATE_EXPC_INF_FR_OPERATE_AC_FR_SALE_G_S8NOTES_PAYABLEADVANCE_RECEIPTST_EQUITY_ATTR_PADVANCE_RECEIPTSC_FR_OTH_INVEST_AINVEST_INCOME9INT_PAYABLEC_PAID_TO_FOR_EMPLC_OUTF_OPERATE_AC_PAID_TO_FOR_EMPLN_INCOME_ATTR_POTH_RECEIV10PAID_IN_CAPITALC_PAID_FOR_DEBTSREVENUEAVAIL_FOR_SALE_FAFOREX_EFFECTSCASH_C_EQUIV11C_PAID_OTH_FINAN_ALT_AMOR_EXPC_PAID_G_SC_PAID_OTH_FINAN_AC_PAID_TO_FOR_EMPLC_OUTF_FR_INVEST_A12N_CF_FR_INVEST_AC_FR_OTH_INVEST_AT_CLC_FR_OTH_INVEST_AC_PAID_FOR_OTH_OP_AADVANCE_RECEIPTS13C_FR_MINO_S_SUBSMINORITY_INTNOPERATE_EXPFOREX_EFFECTSMINORITY_GAININTAN_ASSETS14N_CHANGE_IN_CASHT_NCLC_OUTF_FR_INVEST_AC_INF_FR_OPERATE_AINTAN_ASSETSNOPERATE_INCOME15GAIN_INVESTFIXED_ASSETSMINORITY_INTGAIN_INVESTINT_PAYABLET_EQUITY_ATTR_P16FOREX_EFFECTSC_PAID_INVESTFOREX_EFFECTST_ASSETSC_PAID_OTH_FINAN_ABIZ_TAX_SURCHG17C_FR_OTH_OPERATE_AT_NCLN_CE_END_BALT_NCAC_PAID_FOR_TAXES18N_CF_FR_FINAN_AT_COMPR_INCOMECFSGS_RMINORITY_INTC_PAID_TO_FOR_EMPL19ASSETS_IMPAIR_LOSSAVAIL_FOR_SALE_FAC_OUTF_FR_INVEST_ADILUTED_EPSCOGS20NOPERATE_EXPFIXED_ASSETST_NCAAVAIL_FOR_SALE_FALT_EQUITY_INVEST


## 3.1. Feature Selection Method Based on Machine Learning Model

### 3.1.1. Feature Selection Based on Pearson’s Correlation

Pearson’s correlation coefficient measures the magnitude of a degree of linear correlation, and the greater the absolute value of the correlation coefficient is, the stronger the degree of linear correlation is. The range of the correlation coefficients is found between [-1,1]. Assuming two variables,X and Y, then there are:
(i)
TheX and Y variables are not correlated when the correlation coefficient is 0(ii)
When the values ofX and Y values increase or decrease at the same time, the two variables are positively correlated, and the correlation coefficients are between 0 and 1(iii)
When the value ofX increases and the Y value decreases, or the X value decreases while the Y value increases; the two variables have a negative correlation, with correlation coefficients between-1 and 0Its formula is:(2)ρX⋅Y=∑X‐X¯Y‐Y¯∑X‐X¯2∑Y‐Y¯2.In formula (2), X and Y are two variables, respectively. X¯ and Y¯ denote the mean values of X and Y, respectively.The correlation coefficient is defined only when the standard deviation of both variables is not zero. Scope of application of Pearson’s correlation coefficient:(1)
There is a linear relationship between the two variables, both with continuous data(2)
Overall normal distribution of two variables, or near-normal unimodal distribution(3)
The two variables are in pairs and each pair is independent of each other


### 3.1.2. Feature Selection Based on Lasso

Lasso (least absolute shrinkage and selection operator) is a regression method suitable for multicollinearity problems and can implement feature selection while parameter estimation. The Lasso method is a compression estimation method of reducing the set of variables. It constructs a penalty function that can compress the coefficients of variables and change some regression coefficients to 0, thus achieving the purpose of variable selection [11].Lasso regression is performed by imposing a penalty term on the coefficients of the model, so that some coefficients tend to 0 based on the least-squares estimation, to achieve the purpose of variable selection, and also avoid overfitting, ensure the interpretability and simplicity of the model.Consider the following linear model:(3)Y=Xβ+ε.Y is the vector of n×1, X is the matrix of n×p, and ε is the vector of n×1.β=β1,β2,⋯,βp is the regressive coefficient variable of p×1. Its Lasso is estimated at:
(4)β^=argminβ∥Y−Xβ∥22+λ∑j=1pβjAmong them,λ∑j=1pβj is the penalty term, and λ>0 is the reconciliation parameter, used to control the penalty strength of the model and in turn to control the number of explained variables. When λ is small, the weight ratio of the first part of the upper formula will increase, to minimize the sum of the overall residual squares, so that more explained variables will be added to the model; when λ is large, the weight ratio of the second part of the formula above increases, the regression coefficients of many explanatory variables are compressed and tend to be 0.


### 3.1.3. Multiple Linear Regression Feature Selection

The multiple linear regression model is a model that explains the dependent variable by using two or more explanatory variables. LetY is the dependent variable, X1X2⋯,Xk are k different variables described as explanatory variables. Where X1 is constantly equal to 1, the multiple linear regression model is performed as in Equation (5):
(5)Yi=β1+β2X2i+⋯+βkXki+μii=1,2,⋯,n,where μii=1,2,⋯,n is a random perturbation term; the parameter β1,β2,⋯,βk is called a regression coefficient. Hypothesis,Y=Y1Y2⋮Yn,X=X11X21⋯Xk1X12X22⋯Xk2⋮⋮⋮⋮X1nX2n⋯Xkn,β=β1β2⋮βk,μ=μ1μ2⋮μn. Then, formula (5) is expressed in the matrix form as formula (6).
(6)Y=Xβ+μThe relationship between one or more independent variables is modeled using the least-squares function. Regression analysis in mathematical statistics is a statistical analysis method used to determine the quantitative interdependent relationship between two or more variables. According to the principle of the least-squares method, the estimate ofβi(i=0,1,2,⋯,k), the value of bi(i=0,1,2,⋯,k), should be let by
(7)Q=∑a=1nya−ya∧2=∑a=1nya−b0+b1x1a+b2x2a+⋯+bkxka2⟶min.According to the definition of the least-squares method, in the linear regression model, the estimate of the regression coefficient that minimized the residual sum of squares is called the least-squares-estimation. Equivalent to the smallestβ∧ that makes μ∧′μ∧=Y−Xβ∧′Y−Xβ∧ up, among it, μ′∧ is the transpose of μ∧. To minimize β∧, μ∧′μ∧=Y−Xβ∧′Y−Xβ∧ can be seen as a function about β∧, then the first-order partial derivative of β∧ must be 0, namely, ∂μ∧′μ∧/∂β∧=−2X′Y+2X′Xβ∧=0. Thus, we get equation X′Xβ∧=X′Y, and so β∧=X′X−1X′Y.


### 3.1.4. Feature Selection Based on Random Forest Model

In machine learning, random forest (RF) is a classifier containing multiple decision trees, and the category of its output is determined by the mode of categories output by the individual trees, and the underlying classifier that constitutes the random forest is called the decision tree. The random forest has the advantages of high accuracy, robustness, and easy to use, making it one of the most popular machine learning algorithms today. There are two ways to calculate feature importance in random forest: one method is based on OOB error, called MDA (Mean Decrease Accuracy); another method is based on Gini impurity, called MDG (Mean Decrease Gini). Both methods are the more the numerical decreases and the more important the representation features [12].The MDA is specifically described below:(1)
A random forest model is trained to test the OOB error for each tree in the model using the out-of-bag sample data(2)
The value of the variable v in the out-of-bag sample data is randomly shuffled to retest the OOB error of each tree(3)
The mean of the difference in OOB error for the two tests is the measure of the importance of a single tree to the variable vThe calculation formula is:(8)MDAv=1ntree∑terrOOBt−errOOBt′.The MDG is specifically described below: Gini-based variable importance is measured by the degree of reduced Gini purity due by use of variablev. At classification node t, the Gini coefficient impurity is:
(9)Gt=1−∑K=1Qp2k/,where Q represents the total number of categories for the target variable, pk/trepresents the conditional probability that the target variable is class k in the node t. The Gini nonpurity drop value for each tree was calculated from the formula, and then the results were averaged across all the trees.


### 3.1.5. XGBoost Feature Selection

The extreme gradient lifting algorithm (XGBoost) is a tree-based Boosting algorithm [13]. Compared with the traditional gradient improvement decision tree algorithm, the XGBoost algorithm innovatively uses the second derivative information of the loss function, so that the XGBoost algorithm can converge faster, ensure high solution efficiency, and also increase the scalability. Because a function meets the second-order derivable condition, the function can be used as a custom cost function when appropriate. Another advantage of the XGBoost algorithm is that it borrows the column sampling method in the random forest algorithm and further reduces the computation and overfitting.XGBoost calculates which feature to select as a segmentation point based on the gain of the structure fraction, and the importance of a feature is the sum of its occurrences in all trees. That is, the more times an attribute is used to build a decision tree in a model, the relative importance it is. The XGBoost algorithm can be expressed as:(10)yi^=∑k=1kfkxi,fk∈F,where K indicates the number of trees, F=fx=ωqxq:Rm⟶T,ω∈RTrepresents the function space of the model, and fKxi represents the classification results of the i sample in the K tree. As seen from the expression of the XGBoost algorithm, the model is a collection of iterative residual trees, one tree is added to each iteration, and each tree eventually forms the linear combination of the K tree by learning the residuals of the former N−1 tree.


### 3.1.6. Feature Selection Based on Decision Tree

The structure is similar to that of the tree and consists of oriented edges and nodes. There are three main types of nodes, include the root node, intermediate node, and leaf node. At the top of the decision tree is the root node, which contains the most informative properties; at the bottom of the decision tree are the leaf nodes, representing the results of the classification; between the root and leaf nodes are intermediate nodes used for the testing of feature properties [14].When using the decision tree, the original sample is divided into the training set and the test set, first training a decision tree with the strongest generalization ability, and then predicting using the test set to calculate the generalization error. When training a decision tree, feature selection is about deciding whether to use an indicator as a division basis (as an intermediate node) to help with the classification. The general decision tree algorithm relies on three criteria, namely, information gain, information gain ratio, and Gini index.Information entropy is an indicator used to measure the uncertainty of sample sets, and the more uncertain the set is, the greater the information entropy. Assuming that the proportion of classi samples in sample set D is pii=1,2,3,⋯,y, then the information entropy of sample set D is defined as:
(11)EntD=−∑i=1ypilog2pi.For datasetD, assuming that feature A is selected as the decision tree judgment node, then the information entropy after the action of feature A is defined as:
(12)EntAD=−∑j=1kDjD×EntDj.It was found that decision trees would tend to choose those features with more attribute values when adopting information gain as the criterion for feature selection. To reduce the possible adverse effects of this decision tree preference, the C4.5 decision tree algorithm improves on the original decision tree algorithm. It uses the information gain ratio as a criterion for selecting the optimal partition properties. Information gain ratio is defined as:(13)Gain_rationD,A=GainD,AIVA.Where in equation (13), IVA is known as a fixed value of feature A:
(14)IVA=−∑j=1kDjD×log2DjD.Gini index is also a criterion for a decision tree to select the optimal partition property, representing the probability that a randomly selected sample is misclassification in the set of samples. For sample setD, assuming K categories, the probability of samples belonging to the Kth category is set to A. Then, the Gini index of this probability distribution is:
(15)Ginip=∑k=1kpk1−pk=1−∑k=1kpk2.For the datasetD, its Gini index is:
(16)GiniD=1−∑k=1kCkD2,where Ck represents the number of samples belonging to category k in the sample set D.We have a test at each internal node representation on one attribute by using the tree structure of the decision tree based on the selected features, and test the output on the branches of the tree. Child nodes are generated recursively from top to bottom according to the selected feature evaluation criteria until the decision tree stops growing when the dataset is not separable. The necessary pruning is performed to narrow down the tree structure and alleviate overfitting.


## 3.1.1. Feature Selection Based on Pearson’s Correlation

Pearson’s correlation coefficient measures the magnitude of a degree of linear correlation, and the greater the absolute value of the correlation coefficient is, the stronger the degree of linear correlation is. The range of the correlation coefficients is found between [-1,1]. Assuming two variables,X and Y, then there are:
(i)
TheX and Y variables are not correlated when the correlation coefficient is 0(ii)
When the values ofX and Y values increase or decrease at the same time, the two variables are positively correlated, and the correlation coefficients are between 0 and 1(iii)
When the value ofX increases and the Y value decreases, or the X value decreases while the Y value increases; the two variables have a negative correlation, with correlation coefficients between-1 and 0Its formula is:(2)ρX⋅Y=∑X‐X¯Y‐Y¯∑X‐X¯2∑Y‐Y¯2.In formula (2), X and Y are two variables, respectively. X¯ and Y¯ denote the mean values of X and Y, respectively.The correlation coefficient is defined only when the standard deviation of both variables is not zero. Scope of application of Pearson’s correlation coefficient:(1)
There is a linear relationship between the two variables, both with continuous data(2)
Overall normal distribution of two variables, or near-normal unimodal distribution(3)
The two variables are in pairs and each pair is independent of each other


## 3.1.2. Feature Selection Based on Lasso

Lasso (least absolute shrinkage and selection operator) is a regression method suitable for multicollinearity problems and can implement feature selection while parameter estimation. The Lasso method is a compression estimation method of reducing the set of variables. It constructs a penalty function that can compress the coefficients of variables and change some regression coefficients to 0, thus achieving the purpose of variable selection [11].Lasso regression is performed by imposing a penalty term on the coefficients of the model, so that some coefficients tend to 0 based on the least-squares estimation, to achieve the purpose of variable selection, and also avoid overfitting, ensure the interpretability and simplicity of the model.Consider the following linear model:(3)Y=Xβ+ε.Y is the vector of n×1, X is the matrix of n×p, and ε is the vector of n×1.β=β1,β2,⋯,βp is the regressive coefficient variable of p×1. Its Lasso is estimated at:
(4)β^=argminβ∥Y−Xβ∥22+λ∑j=1pβjAmong them,λ∑j=1pβj is the penalty term, and λ>0 is the reconciliation parameter, used to control the penalty strength of the model and in turn to control the number of explained variables. When λ is small, the weight ratio of the first part of the upper formula will increase, to minimize the sum of the overall residual squares, so that more explained variables will be added to the model; when λ is large, the weight ratio of the second part of the formula above increases, the regression coefficients of many explanatory variables are compressed and tend to be 0.


## 3.1.3. Multiple Linear Regression Feature Selection

The multiple linear regression model is a model that explains the dependent variable by using two or more explanatory variables. LetY is the dependent variable, X1X2⋯,Xk are k different variables described as explanatory variables. Where X1 is constantly equal to 1, the multiple linear regression model is performed as in Equation (5):
(5)Yi=β1+β2X2i+⋯+βkXki+μii=1,2,⋯,n,where μii=1,2,⋯,n is a random perturbation term; the parameter β1,β2,⋯,βk is called a regression coefficient. Hypothesis,Y=Y1Y2⋮Yn,X=X11X21⋯Xk1X12X22⋯Xk2⋮⋮⋮⋮X1nX2n⋯Xkn,β=β1β2⋮βk,μ=μ1μ2⋮μn. Then, formula (5) is expressed in the matrix form as formula (6).
(6)Y=Xβ+μThe relationship between one or more independent variables is modeled using the least-squares function. Regression analysis in mathematical statistics is a statistical analysis method used to determine the quantitative interdependent relationship between two or more variables. According to the principle of the least-squares method, the estimate ofβi(i=0,1,2,⋯,k), the value of bi(i=0,1,2,⋯,k), should be let by
(7)Q=∑a=1nya−ya∧2=∑a=1nya−b0+b1x1a+b2x2a+⋯+bkxka2⟶min.According to the definition of the least-squares method, in the linear regression model, the estimate of the regression coefficient that minimized the residual sum of squares is called the least-squares-estimation. Equivalent to the smallestβ∧ that makes μ∧′μ∧=Y−Xβ∧′Y−Xβ∧ up, among it, μ′∧ is the transpose of μ∧. To minimize β∧, μ∧′μ∧=Y−Xβ∧′Y−Xβ∧ can be seen as a function about β∧, then the first-order partial derivative of β∧ must be 0, namely, ∂μ∧′μ∧/∂β∧=−2X′Y+2X′Xβ∧=0. Thus, we get equation X′Xβ∧=X′Y, and so β∧=X′X−1X′Y.


## 3.1.4. Feature Selection Based on Random Forest Model

In machine learning, random forest (RF) is a classifier containing multiple decision trees, and the category of its output is determined by the mode of categories output by the individual trees, and the underlying classifier that constitutes the random forest is called the decision tree. The random forest has the advantages of high accuracy, robustness, and easy to use, making it one of the most popular machine learning algorithms today. There are two ways to calculate feature importance in random forest: one method is based on OOB error, called MDA (Mean Decrease Accuracy); another method is based on Gini impurity, called MDG (Mean Decrease Gini). Both methods are the more the numerical decreases and the more important the representation features [12].The MDA is specifically described below:(1)
A random forest model is trained to test the OOB error for each tree in the model using the out-of-bag sample data(2)
The value of the variable v in the out-of-bag sample data is randomly shuffled to retest the OOB error of each tree(3)
The mean of the difference in OOB error for the two tests is the measure of the importance of a single tree to the variable vThe calculation formula is:(8)MDAv=1ntree∑terrOOBt−errOOBt′.The MDG is specifically described below: Gini-based variable importance is measured by the degree of reduced Gini purity due by use of variablev. At classification node t, the Gini coefficient impurity is:
(9)Gt=1−∑K=1Qp2k/,where Q represents the total number of categories for the target variable, pk/trepresents the conditional probability that the target variable is class k in the node t. The Gini nonpurity drop value for each tree was calculated from the formula, and then the results were averaged across all the trees.


## 3.1.5. XGBoost Feature Selection

The extreme gradient lifting algorithm (XGBoost) is a tree-based Boosting algorithm [13]. Compared with the traditional gradient improvement decision tree algorithm, the XGBoost algorithm innovatively uses the second derivative information of the loss function, so that the XGBoost algorithm can converge faster, ensure high solution efficiency, and also increase the scalability. Because a function meets the second-order derivable condition, the function can be used as a custom cost function when appropriate. Another advantage of the XGBoost algorithm is that it borrows the column sampling method in the random forest algorithm and further reduces the computation and overfitting.XGBoost calculates which feature to select as a segmentation point based on the gain of the structure fraction, and the importance of a feature is the sum of its occurrences in all trees. That is, the more times an attribute is used to build a decision tree in a model, the relative importance it is. The XGBoost algorithm can be expressed as:(10)yi^=∑k=1kfkxi,fk∈F,where K indicates the number of trees, F=fx=ωqxq:Rm⟶T,ω∈RTrepresents the function space of the model, and fKxi represents the classification results of the i sample in the K tree. As seen from the expression of the XGBoost algorithm, the model is a collection of iterative residual trees, one tree is added to each iteration, and each tree eventually forms the linear combination of the K tree by learning the residuals of the former N−1 tree.


## 3.1.6. Feature Selection Based on Decision Tree

The structure is similar to that of the tree and consists of oriented edges and nodes. There are three main types of nodes, include the root node, intermediate node, and leaf node. At the top of the decision tree is the root node, which contains the most informative properties; at the bottom of the decision tree are the leaf nodes, representing the results of the classification; between the root and leaf nodes are intermediate nodes used for the testing of feature properties [14].When using the decision tree, the original sample is divided into the training set and the test set, first training a decision tree with the strongest generalization ability, and then predicting using the test set to calculate the generalization error. When training a decision tree, feature selection is about deciding whether to use an indicator as a division basis (as an intermediate node) to help with the classification. The general decision tree algorithm relies on three criteria, namely, information gain, information gain ratio, and Gini index.Information entropy is an indicator used to measure the uncertainty of sample sets, and the more uncertain the set is, the greater the information entropy. Assuming that the proportion of classi samples in sample set D is pii=1,2,3,⋯,y, then the information entropy of sample set D is defined as:
(11)EntD=−∑i=1ypilog2pi.For datasetD, assuming that feature A is selected as the decision tree judgment node, then the information entropy after the action of feature A is defined as:
(12)EntAD=−∑j=1kDjD×EntDj.It was found that decision trees would tend to choose those features with more attribute values when adopting information gain as the criterion for feature selection. To reduce the possible adverse effects of this decision tree preference, the C4.5 decision tree algorithm improves on the original decision tree algorithm. It uses the information gain ratio as a criterion for selecting the optimal partition properties. Information gain ratio is defined as:(13)Gain_rationD,A=GainD,AIVA.Where in equation (13), IVA is known as a fixed value of feature A:
(14)IVA=−∑j=1kDjD×log2DjD.Gini index is also a criterion for a decision tree to select the optimal partition property, representing the probability that a randomly selected sample is misclassification in the set of samples. For sample setD, assuming K categories, the probability of samples belonging to the Kth category is set to A. Then, the Gini index of this probability distribution is:
(15)Ginip=∑k=1kpk1−pk=1−∑k=1kpk2.For the datasetD, its Gini index is:
(16)GiniD=1−∑k=1kCkD2,where Ck represents the number of samples belonging to category k in the sample set D.We have a test at each internal node representation on one attribute by using the tree structure of the decision tree based on the selected features, and test the output on the branches of the tree. Child nodes are generated recursively from top to bottom according to the selected feature evaluation criteria until the decision tree stops growing when the dataset is not separable. The necessary pruning is performed to narrow down the tree structure and alleviate overfitting.


## 3.2. Experiments of Important Feature Selection

The training set and the test set are divided by using the train_test_split method of model_selection in machine learning sklearn; the ratio of the training set to the test set is determined as 7 : 3. To prevent the sample division from affecting the prediction results, random seeds are set when dividing the training set and the test sets. “FLAG” serves as a label column, and other indicators serve as feature columns. Using the data from the training set, for the Pearson correlation coefficient method, all correlation indicators and label columns are imported to find the top 20 features for the correlations with the label column; For the Lasso method, a total of nonzero16 metrics with a weight coefficient are found; the top 20 features are ranked on the model using all correlation metrics to find the top features, respectively. Features are ranked using feature importance on the LR, RF, XGBoost, and DT models, respectively, finding the top 20 features. The main purpose is to obtain the top 20 attributes using sklearn’s ranking of feature_importances_ attribute values. The inherent mechanism of DT is to discriminate the effect of features on the purity increase of nodes based on the actual discriminating criteria such as GINI, information entropy, and information entropy gain. The parameters used on LR, RF, XGBoost, DT models for indicators selection are shown in Table3. These two algorithms of Pearson’s and Lasso do not use parameters. Table 3 shows the names and values of these parameters.Table 3
Parameters list of machine learning model.
NoModel nameParametersNameValue1LRpenaltyL1C1.0solverliblinearmulti_classover2RFn_estimators10max_features20splitterbest3XGBoostboostergbtreemax_depth6learning_rate0.34DTn_estimators10splitterbestmax_depth3Finally, through experiments, the important indicators related to financial fraud are selected by each algorithm and model, as shown in Table4.Table 4
Selection table of important indicators based on machine learning methods.
NumModelPearsonLassoLRRFXGBoostDT1ARINT_PAYABLET_LIAB_EQUITYINT_PAYABLET_ASSETSAVAIL_FOR_SALE_FA2PREPAYMENTT_LIABT_PROFITT_LIABT_LIABT_LIAB3OTH_RECEIVC_OUTF_FR_INVEST_AT_COGSC_OUTF_FR_INVEST_AN_CE_END_BALINT_PAYABLE4AVAIL_FOR_SALE_FAINTAN_ASSETSN_CF_FR_INVEST_AINTAN_ASSETSAVAIL_FOR_SALE_FAN_CE_END_BAL5CIPLT_EQUITY_INVESTN_CF_OPERATE_ALT_EQUITY_INVESTLT_EQUITY_INVESTT_REVENUE6DEFER_TAX_ASSETSDILUTED_EPSN_CF_FR_FINAN_ADILUTED_EPSREVENUEC_OUTF_FR_FINAN_A7OTH_NCANOPERATE_EXPN_INCOMENOPERATE_EXPC_INF_FR_OPERATE_AC_FR_SALE_G_S8NOTES_PAYABLEADVANCE_RECEIPTST_EQUITY_ATTR_PADVANCE_RECEIPTSC_FR_OTH_INVEST_AINVEST_INCOME9INT_PAYABLEC_PAID_TO_FOR_EMPLC_OUTF_OPERATE_AC_PAID_TO_FOR_EMPLN_INCOME_ATTR_POTH_RECEIV10PAID_IN_CAPITALC_PAID_FOR_DEBTSREVENUEAVAIL_FOR_SALE_FAFOREX_EFFECTSCASH_C_EQUIV11C_PAID_OTH_FINAN_ALT_AMOR_EXPC_PAID_G_SC_PAID_OTH_FINAN_AC_PAID_TO_FOR_EMPLC_OUTF_FR_INVEST_A12N_CF_FR_INVEST_AC_FR_OTH_INVEST_AT_CLC_FR_OTH_INVEST_AC_PAID_FOR_OTH_OP_AADVANCE_RECEIPTS13C_FR_MINO_S_SUBSMINORITY_INTNOPERATE_EXPFOREX_EFFECTSMINORITY_GAININTAN_ASSETS14N_CHANGE_IN_CASHT_NCLC_OUTF_FR_INVEST_AC_INF_FR_OPERATE_AINTAN_ASSETSNOPERATE_INCOME15GAIN_INVESTFIXED_ASSETSMINORITY_INTGAIN_INVESTINT_PAYABLET_EQUITY_ATTR_P16FOREX_EFFECTSC_PAID_INVESTFOREX_EFFECTST_ASSETSC_PAID_OTH_FINAN_ABIZ_TAX_SURCHG17C_FR_OTH_OPERATE_AT_NCLN_CE_END_BALT_NCAC_PAID_FOR_TAXES18N_CF_FR_FINAN_AT_COMPR_INCOMECFSGS_RMINORITY_INTC_PAID_TO_FOR_EMPL19ASSETS_IMPAIR_LOSSAVAIL_FOR_SALE_FAC_OUTF_FR_INVEST_ADILUTED_EPSCOGS20NOPERATE_EXPFIXED_ASSETST_NCAAVAIL_FOR_SALE_FALT_EQUITY_INVEST


## 4. Key Indicators Selection Method of Financial Fraud Prediction Model Based on Machine Learning Hybrid Mode

### 4.1. Intermediate Evaluation Indicator

In this paper, the performance measures of the classification model adopt the confusion matrix and True Positive Rate, False Positive Rate, and AUC. In the binary classification problem, the confusion matrix is a second-order matrix. Among them, this paper uses normal samples as positive (0) and fake samples as negative (1), usingTPFPTNFN to indicate four cases of whether the classifier predicted or correct on the dataset.TP: True Positive, predict positive class to be positive;FP: False Positive, predict negative class to be positive;TN: True Negative, predict negative class to be negative;FN: False Negative, predict positive class to be negative;The representation in the confusion matrix is shown in Table5.Table 5
Confusion matrix.
TruePredict010TPFN1FPTNTPR (True Positive Rate) represents the proportion of true classes in the samples predicted to be positive. The calculation formula is:(17)TPR=TPTP+FN.FPR( False Positive Rate) represents the proportion of true classes in samples predicted to be negative. The calculation formula is:(18)FPR=FPFP+TN.The AUC is the area under the ROC curve, giving the average performance value of the classifier. AUC values can be used to evaluate the effect of binary classification problem. The closer the AUC is to 1.0, the higher the authenticity of the detection method. The calculation formula is:(19)AUC=1+TPR‐FPR2.


### 4.2. Key Indicators Selection Method Based on Machine Learning Hybrid Mode

Features are ranked using feature importance on the Pearson, Lasso, LR, RF, XGBoost, and DT models, respectively, to find the top 20 features. In order to prevent the homogeneous model from restricting the selection of indicators, the selected intermediate model should be heterogeneous with the above six models. The support vector machine (SVM) with fewer parameters and good effects is used as the intermediate model to conduct AUC tests on the indicators selected by various models. Specifically, for the features selected by each model, the corresponding data in the feature columns of the training set are, respectively, selected as the feature columns of the new training set, and imported into the support vector machine model for retraining. The data corresponding to the test set are selected as a new test set feature column for prediction.We first test the values of the AUC on the support vector machines for the features selected by the Pearson, Lasso, RF, XGBoost, DT, LR. The experimental results are shown in Figure4:Figure 4
AUC comparison result of individual feature selection methods on the SVM.The models and algorithms used in this paper are divided into three categories, include tree model (DT, RF, XGBoost), algorithm (Lasso, Pearson), and basic model (Logistic). The models with the first AUC ranking are selected from these three categories, and the final models selected are RF, Lasso, and Logistic. Then, these three models with good performance are mixed. In this paper, the network search mechanism is used to exhaustively list all the combined models, so that the optimal hybrid model can be compared easily. Finally, all the hybrid models are formed as RF+ Lasso, Lasso+ Logistic, RF+ Logistic, and RF+ Lasso+ Logistic. The repeated features in these hybrid models are selected, and finally these hybrid models are tested again on the support vector machine for AUC values. The experimental results are shown in Figure5.Figure 5
AUC comparison result of repeated features selected by hybrid models on the SVM.The experimental results in Figure5 show that the features jointly selected by Lasso and the RF performed best on the SVM. Therefore, the corresponding selected indicators are shown in Table 6.Table 6
Important indicators selected on the optimal hybrid model.
NumFeatureNumFeature1INT_PAYABLE5DILUTED_EPS2C_OUTF_FR_INVEST_A6NOPERATE_EXP3INTAN_ASSETS7ADVANCE_RECEIPTS4LT_EQUITY_INVEST8C_PAID_TO_FOR_EMPL


### 4.3. Correlation Analysis of the Key Indicators

Due to the internal logic of the financial statement data itself, there is also some degree of autocorrelation between the key features selected in this paper, which will affect the accuracy of the model estimate. Therefore, before substituting into the model, we first have to analyze multicollinearity and remove the features with high partial autocorrelation. We use autocorrelation to conduct experiments on the explanatory power of the features in the table described above and to analyze the multicollinearity problem of these features. After autocorrelation analysis, we remove features with autocorrelation greater than the threshold.Correlation coefficients are the amount of the degree of linear correlation between the studied variables, generally indicated by the letterr. Due to the different subjects, the correlation coefficient is defined in many ways, including the Pearson correlation coefficient. The correlation is considered strong if the absolute value of the correlation coefficient r for A and B is above 0.7; correlation coefficient r between 0.3 and 0.7, correlation is weak; correlation coefficient r below 0.3, no correlation. In this paper, the Pearson coefficient method is used to calculate the linear correlation coefficient between the two features. The corresponding calculation formula is:
(20)ρX⋅Y=N∑XY−∑X∑YN∑X2−∑X2N∑Y2−∑Y2.The correlation experiment is performed on the 8 indicators selected in Table6, and the correlation between the indicators is calculated by formula (20). The experimental results are shown in Figure 6.Figure 6
Correlation figure of the key indicators.As is shown in Figure6, the autocorrelation value for both features “INTAN_ASSETS” and “C_PAID_TO_FOR_EMPL” is 0.76, greater than the set threshold of 0.7. Therefore, these two features belong to the strong correlation feature, and considering the correlation of the features, the “C_PAID_TO_FOR_EMPL” feature is deleted. Final key indicators for the prediction model are shown in Table 7.Table 7
The final key indicators.
NumFeature1INT_PAYABLE2C_OUTF_FR_INVEST_A3INTAN_ASSETS4LT_EQUITY_INVEST5DILUTED_EPS6NOPERATE_EXP7ADVANCE_RECEIPTS


## 4.1. Intermediate Evaluation Indicator

In this paper, the performance measures of the classification model adopt the confusion matrix and True Positive Rate, False Positive Rate, and AUC. In the binary classification problem, the confusion matrix is a second-order matrix. Among them, this paper uses normal samples as positive (0) and fake samples as negative (1), usingTPFPTNFN to indicate four cases of whether the classifier predicted or correct on the dataset.TP: True Positive, predict positive class to be positive;FP: False Positive, predict negative class to be positive;TN: True Negative, predict negative class to be negative;FN: False Negative, predict positive class to be negative;The representation in the confusion matrix is shown in Table5.Table 5
Confusion matrix.
TruePredict010TPFN1FPTNTPR (True Positive Rate) represents the proportion of true classes in the samples predicted to be positive. The calculation formula is:(17)TPR=TPTP+FN.FPR( False Positive Rate) represents the proportion of true classes in samples predicted to be negative. The calculation formula is:(18)FPR=FPFP+TN.The AUC is the area under the ROC curve, giving the average performance value of the classifier. AUC values can be used to evaluate the effect of binary classification problem. The closer the AUC is to 1.0, the higher the authenticity of the detection method. The calculation formula is:(19)AUC=1+TPR‐FPR2.


## 4.2. Key Indicators Selection Method Based on Machine Learning Hybrid Mode

Features are ranked using feature importance on the Pearson, Lasso, LR, RF, XGBoost, and DT models, respectively, to find the top 20 features. In order to prevent the homogeneous model from restricting the selection of indicators, the selected intermediate model should be heterogeneous with the above six models. The support vector machine (SVM) with fewer parameters and good effects is used as the intermediate model to conduct AUC tests on the indicators selected by various models. Specifically, for the features selected by each model, the corresponding data in the feature columns of the training set are, respectively, selected as the feature columns of the new training set, and imported into the support vector machine model for retraining. The data corresponding to the test set are selected as a new test set feature column for prediction.We first test the values of the AUC on the support vector machines for the features selected by the Pearson, Lasso, RF, XGBoost, DT, LR. The experimental results are shown in Figure4:Figure 4
AUC comparison result of individual feature selection methods on the SVM.The models and algorithms used in this paper are divided into three categories, include tree model (DT, RF, XGBoost), algorithm (Lasso, Pearson), and basic model (Logistic). The models with the first AUC ranking are selected from these three categories, and the final models selected are RF, Lasso, and Logistic. Then, these three models with good performance are mixed. In this paper, the network search mechanism is used to exhaustively list all the combined models, so that the optimal hybrid model can be compared easily. Finally, all the hybrid models are formed as RF+ Lasso, Lasso+ Logistic, RF+ Logistic, and RF+ Lasso+ Logistic. The repeated features in these hybrid models are selected, and finally these hybrid models are tested again on the support vector machine for AUC values. The experimental results are shown in Figure5.Figure 5
AUC comparison result of repeated features selected by hybrid models on the SVM.The experimental results in Figure5 show that the features jointly selected by Lasso and the RF performed best on the SVM. Therefore, the corresponding selected indicators are shown in Table 6.Table 6
Important indicators selected on the optimal hybrid model.
NumFeatureNumFeature1INT_PAYABLE5DILUTED_EPS2C_OUTF_FR_INVEST_A6NOPERATE_EXP3INTAN_ASSETS7ADVANCE_RECEIPTS4LT_EQUITY_INVEST8C_PAID_TO_FOR_EMPL


## 4.3. Correlation Analysis of the Key Indicators

Due to the internal logic of the financial statement data itself, there is also some degree of autocorrelation between the key features selected in this paper, which will affect the accuracy of the model estimate. Therefore, before substituting into the model, we first have to analyze multicollinearity and remove the features with high partial autocorrelation. We use autocorrelation to conduct experiments on the explanatory power of the features in the table described above and to analyze the multicollinearity problem of these features. After autocorrelation analysis, we remove features with autocorrelation greater than the threshold.Correlation coefficients are the amount of the degree of linear correlation between the studied variables, generally indicated by the letterr. Due to the different subjects, the correlation coefficient is defined in many ways, including the Pearson correlation coefficient. The correlation is considered strong if the absolute value of the correlation coefficient r for A and B is above 0.7; correlation coefficient r between 0.3 and 0.7, correlation is weak; correlation coefficient r below 0.3, no correlation. In this paper, the Pearson coefficient method is used to calculate the linear correlation coefficient between the two features. The corresponding calculation formula is:
(20)ρX⋅Y=N∑XY−∑X∑YN∑X2−∑X2N∑Y2−∑Y2.The correlation experiment is performed on the 8 indicators selected in Table6, and the correlation between the indicators is calculated by formula (20). The experimental results are shown in Figure 6.Figure 6
Correlation figure of the key indicators.As is shown in Figure6, the autocorrelation value for both features “INTAN_ASSETS” and “C_PAID_TO_FOR_EMPL” is 0.76, greater than the set threshold of 0.7. Therefore, these two features belong to the strong correlation feature, and considering the correlation of the features, the “C_PAID_TO_FOR_EMPL” feature is deleted. Final key indicators for the prediction model are shown in Table 7.Table 7
The final key indicators.
NumFeature1INT_PAYABLE2C_OUTF_FR_INVEST_A3INTAN_ASSETS4LT_EQUITY_INVEST5DILUTED_EPS6NOPERATE_EXP7ADVANCE_RECEIPTS


## 5. Conclusion

This paper mainly analyzes the selection method of key indicators in the financial fraud prediction model, and proposes the key indicators selection method based on the machine learning hybrid model because of the existing feature selection method with low prediction accuracy. First for the dataset, preprocessing includes missing value processing and standardization of the data. The feature selection methods in multiple machine learning models are then described and the selected top 20 features for each model. Support vector machine is used as the intermediate model for AUC testing, the top models are combined, and the repeated features in the hybrid model are selected as the pre-selection features. Tested the selected pre-selected features, and the model combined with the highest AUC values is selected. Finally, through the correlation experiment of the indicators, the final key indicators are obtained. The novel key indicators selection method based on machine learning hybrid model is proposed here in this paper and effectively improves the prediction accuracy. This novel key indicators method provides an important basis for the construction of the financial fraud prediction model. However, the number and types of models for feature selection in the experiment are large, and the optimization of all parameters will cause a large time complexity, thus affecting the experimental effect. Therefore, the default parameters are used in this paper. Future work will focus on parameter optimization of the models and building a financial fraud prediction model by using the selected key indicators.



---

*Source: 6542652-2022-03-28.xml*