# Extreme Learning Machine Enhanced Gradient Boosting for Credit Scoring
## 3 Experimental Settings
### Finetuning Process of Credit Scoring Benchmarks  
LR is a simple and linear algorithm that has been widely pursued in practical credit scoring for banks and lending institutions. To get a good predictive performance of LR, we adopt [PERSON]'s method to iteratively minimize the empirical credit scoring error, \(L_{1}\) and \(L_{2}\) penalty is further introduced to alleviate the overfitting problem, which is optimized from the value set of \(\{0.01,0.1,1,10,100\}\). KNN is a classical machine learning method that achieves credit scoring by finding the \(K\) nearest samples. Its final predictions are averaged from the states of the searched k nearest samples. Therefore, we focus on finetuning this parameter from the interval of \([5,20]\) with a search stride of 1. To accelerate the finetuning process of SVM, in this study, radial basis function (RBF) kernel-based SVM is employed in this work for credit scoring. The major hyper-parameters that need to be finetuned in a RBF kernel-based SVM are \(\gamma\) and \(C\), where \(\gamma\) regulates the form of mapping space and \(C\) represents penalty coefficient, which quantifies the penalty degree of misclassified samples. We pre-fix the entire structure with two hidden layers to implement NN. We begin by constructing a stable NN structure from the architectural set with \(64,128,\) and \(256\) hidden nodes for hidden layers. Following that, we optimize the learning rate based on an \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Hyperparameters** & **Description** & **Searching Space** & **Stride** \\ \hline n\_estimators & The number of DTs & \(100\) & / \\   learning\_rate & Learning rate that control & \(0.001,0.2\) & biSection \\  & the optimization step & \(0.001,0.2\) & biSection \\   max\_depth & Maximum depth of & \(3,11\) & \(1\) \\  & DT in each & \(3,11\) & \(1\) \\ min\_sample\_split & Minimal samples to split & \(20,100\) & \(5\) \\  & at each splitting node & \(20,100\) & \(5\) \\   min\_samples\_leaf & Minimal samples at & \(20,100\) & \(5\) \\  & each leaf node & \(20,100\) & \(5\) \\   subsample & Undersampling ratio on & \(0.6,1\) & \(0.05\) \\  & the training set & \(0.6,1\) & \(0.05\) \\   max\_features & Maximum candidate features & \(\sqrt{m}\) & biSection \\  & for node splitting & \(0\) & \(64,128,256\) & / \\  & Number of hidden nodes of & \(64,128,256\) & / \\  & ELM & \(64,128,256\) & / \\  & Period of performing feature & \(5,10\) & / \\ \(n_{A}\) & augmentation for boosting & & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Parameters setting of AugBoost-ELM. initialized searching space of \([0.0001,0.5]\). To finetune the learning rate, we first select an acceptable learning rate interval using dichotomy and then identify the ideal learning rate using grid search with 5-fold cross-validation from the preset learning rate interval. Each hidden layer is activated by a ReLU function, and a sigmoid activation function is followed to get the probabilistic predictions. Because the credit scoring process is modeled as a binary classification process, the weight parameters of the NN are optimized using a binary cross-entropy loss function. Additionally, dropout is employed to minimize overfitting, and the optimal dropout rate is searched from the range of \([0.1,0.5]\) with an optimization step of 0.1. RF is an efficient approach that realizes credit scoring in a Bagging-ensemble way. In the finetuning of a RF, we first searched the parameter that denotes the number of DTs to ensemble a RF from the interval of \([100,500]\) with a search step 100. After determining the overall ensemble framework of RF, we further finetune the parameters in each DT. To encourage the diversity of the ensemble, RF allows each tree within it to grow a deep structure to accommodate high bias low variance predictions. we first finetune the maximum depth of each DT in RF from the interval of \(\{10,15,**\}\) with a finetune step of 1, where \(**\) denotes a DT can grow its structure with an arbitrary depth.