# Extreme Learning Machine Enhanced Gradient Boosting for Credit Scoring
## 2 Methodology  
### Extreme Learning Machine  
Extreme learning machine (ELM) is a new fast learning algorithm [42]. Given an arbitrary credit dataset \(\{(\mathbf{x}_{1},\mathbf{y}_{1}),(\mathbf{x}_{2},\mathbf{y}_{2}),\cdots,( \mathbf{x}_{N},\mathbf{y}_{N})\}\), where \(\mathbf{x}_{j}=[x_{j1},x_{j2},\cdots,x_{jm}]\), \(m\) is the number of features, \(\mathbf{y}_{j}=[y_{j1},y_{j2},\cdots,y_{jC}]\in R^{C}\) is the label of \(i\)-th sample in a one-hot form, and \(C\) is the number of classes. For a single-layer feed-forward neural network with \(L\) hidden nodes, it can be expressed as:  
\[\sum_{i=1}^{L}\beta_{iS}(\mathbf{W}_{i}\mathbf{x}_{j}+b_{i})=\mathbf{o}_{j},j= 1,2,\cdots,N, \tag{1}\]  
where \(g(x)\) is the Gaussian radial basis activation function, \(\mathbf{W}_{i}=[w_{i1},w_{i2},\cdots,w_{im}]^{T}\)is the weight vector of the \(i\)-th neuron in the input layer, \(\mathbf{\beta}=[\beta_{i1},\beta_{i2},\cdots,\beta_{iC}]^{T}\) is the output weight vector of the \(i\)-th neuron in the hidden layer, and \(\beta_{ic}\) represents the connection weight between the \(i\)-th neuron in the hidden layer and the \(c\)-th neuron in the output layer. \(b_{i}\) is the bias of the \(i\)-th neuron in the hidden layer. \(\mathbf{W}_{i}\mathbf{x}_{j}\) calculates the inner product of the input weight vector and input vector. The goal of ML-based credit scoring is to minimize empirical risk. In the credit scoring based on a single hidden layer neural network, the goal of the learning process is to minimize the output error, which can be expressed as:  
\[\sum_{j=1}^{N}\|\mathbf{o}_{j}-\mathbf{y}_{j}\|=0,j=1,2,\cdots,N. \tag{2}\]  
That is, there are \(\mathbf{W}_{i}\mathbf{x}_{j}\), and \(b_{i}\), such that:  
\[\sum_{i=1}^{L}\beta_{iS}(\mathbf{W}_{i}\mathbf{x}_{j}+b_{i})=\mathbf{y}_{j},j= 1,2,\cdots,N. \tag{3}\]  
We matrix Equation (3) into:  
\[\mathbf{H}\mathbf{\beta}=\mathbf{Y}, \tag{4}\]  
where \(\mathbf{H}\) represents the output matrix of the hidden layer and \(\mathbf{\beta}\) is the connection weight between the hidden layer and the output layer, \(\mathbf{Y}\) denotes the expected output matrix. Specifically, Equation (4) can be expressed as:  
\[\mathbf{H}(\mathbf{W}_{1},\mathbf{W}_{2},\cdots,\mathbf{W}_{L};b_ {1},b_{2},\cdots,b_{L};\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{N})\] \[= \left[ \begin{array}{ccc}g(\mathbf{W}_{1}\mathbf{x}_{1}+b_{1})& \cdots&\mathbf{W}_{L}\mathbf{x}_{1}+b_{L}\\ \vdots&\cdots&\vdots\\ g(\mathbf{W}_{1}\mathbf{x}_{N}+b_{1})&\cdots&g(\mathbf{W}_{L}\mathbf{x}_{N}+b_ {L})\end{array} \right]\]  
where \(\mathbf{\beta}=[\mathbf{\beta}_{1}^{T},\cdots,\mathbf{\beta}_{L}^{T}]^{T}\), \(\mathbf{Y}=[\mathbf{y}_{1}^{T},\cdots,\mathbf{y}_{L}^{T}]^{T}\). ELM-based credit scoring aims to minimize the empirical risk, Therefore, we expect to find \(\hat{\mathbf{W}}_{i}\hat{b}_{i}\hat{\beta}_{i}\), and make:  
\[\|\mathbf{H}(\hat{\mathbf{W}}_{i};\hat{b}_{i})\hat{\beta}_{i}-\mathbf{Y}\|= \underset{\mathbf{W},\mathbf{\beta}}{min}\|\mathbf{H}(\mathbf{W}_{i};b_{i})\mathbf{ \beta}_{i}-\mathbf{Y}\|. \tag{5}\]  
This is equivalent to minimizing the loss function:\[\mathcal{E}=\sum_{j=1}^{N}(\sum_{i=1}^{L}\beta_{i;\beta}(\mathbf{W}_{i}\mathbf{x}_ {j}+b_{i})-\mathbf{y}_{j})^{2}. \tag{6}\]  
Different from the traditional neural network that minimizes loss function based on gradient descent algorithm, ELM does not involve an error back-propagation process for weight update. In the implementation of ELM, the input weight \(\mathbf{W}_{i}\) and the bias of the hidden layer are randomly initialized and do not update in the whole training process. The output matrix \(\mathbf{H}\) of the hidden layer of ELM is uniquely determined in the training process. Therefore, the training single hidden layer neural network can be transformed into finding a solution of linear equation \(\mathbf{H}\beta=\mathbf{Y}\), so as to find the optimal \(\beta\) to minimize the training error.