# Extreme Learning Machine Enhanced Gradient Boosting for Credit Scoring
## 2 Methodology
### GBDT Enhanced by Supervised Extreme Learning Machine  
GBDT realizes ensemble by iteratively modifying the training target while keeping the training features unchanged. This optimization method differs greatly from the integration strategy of random forest. Random forest integrates multiple DTs by keeping the training target unchanged while the diversity of each DT in RF is enhanced by modifying the input samples based on the Bagging algorithm. [PERSON] [39] proposed three methods to step-wise augmented GBDT, which borrowed the advantage of RF-type ensemble strategy and the training pattern of boosting-type ensemble approaches. In [PERSON]'s work, random projection (RP) [43], PCA is considered as two unsupervised augmented algorithms for enhancing GBDT, and NN is regarded as the supervised augmented method to realize supervised augmented GBDT. [PERSON]'s [40] study has shown promising evidence of supervised augmented GBDT outperforms two unsupervised augmented GBDTs. Though NN-based supervised feature augmentation for GBDT achieved better performance than RP-based and PCA-based unsupervised feature augmentation for GBDT, NN is an algorithm that is optimized based on error backpropagation, which is the related chain principle. Such optimization pattern is inefficient and easily prone to over-fitting as well as easy falls into local optima, leading to the feature augmentation process being a time-consuming process and making the supervised augmented GBDT a complex model to realize fast credit scoring [44]. Based on the above considerations, in this paper, we develop an efficient forward NN algorithm, ELM, to step-wisely augment the GBDT framework, which is AugBoost-ELM. In [PERSON]'s [40] work, three AugBoost models have been developed to enhance the diversity of base learners of GBDT, of which supervised augmented GBDT, AugBoost-NN has been proved to be a more precise classification approach. However, NN-based feature augmentation is based on the training of a good NN framework, while the training process of determining an optimal NN is complex. Besides, considering GBDT is a sequential ensemble framework, the integration of NN for feature augmentation and boosting training patter of GBDT makes AugBoost-NN a training-inefficient algorithm for credit scoring. Therefore, a more efficient augmented GBDT is proposed in this study to enhance the diversity of base learners in GBDT to improve the performance of credit scoring. Figure 1 shows the training process of AugBoost-ELM. As shown in Figure 1, the same as the training pattern of classical GBDT, original features are considered as the input of the base regression DTs. In the training process of AugBoost-ELM, before training a new DT to the further ensemble model, each original feature is augmented by a fast neural network-ELM, and the augmented features and the original features are combined as the input of each base learner. By doing this, different from the input of each DT in the original GBDT, the input of each DT of AugBoost-ELM is diversified, thus leading to the base learners of AugBoost-ELM being more diversified and driving the ensemble models to make diversified prediction results. The training pseudo-code of AugBoost-ELM is shown in Algorithm 1. ```
0: A credit training set \(\{(\mathbf{x}_{1},y_{1}),(\mathbf{x}_{2},y_{2}),\cdots\ (\mathbf{x}_{N},y_{N})\}\)
0: An augmented GDBT \(F_{T}(\mathbf{x})\) which performs step-wise feature augmentation with ELMs
1: Initialize first regression DT \(F_{0}(\mathbf{x})\) as: \(\arg\min\sum_{i=1}^{N}L(y_{i},\rho),\rho\) is the optimization step of \(F_{T}(\mathbf{x})\), and the training targets of \(\{y_{i}\}_{i=1}^{N}\). 2:for\(t=1,2,\cdots,T\)do
3:if\((t-1)^{\rho}\omega_{A}=0\)then
4: Randomly split original training feature set into \(K\) random feature subsets. 5:for\(k=1,2,\cdots,K\)do
6: Randomly select a feature subset with selection operation \(S_{t,k}\)
7: Training an ELM as an augmentation function \(f_{t,k}\) to perform feature augmentation for GBDT.