# Advanced Default Risk Prediction in Small and Medium-Sized Enterprises Using Large Language Models
## 4 Experiments and Results
### Machine Learning Methodology
#### 4.2.2 Results  
The experiment incorporated diverse datasets and machine learning algorithms for training and evaluation. The evaluation metrics were computed using a weighted average approach, accounting for the two-label distribution in the dataset. Detailed results are presented in Table 7.  
Initially, the machine learning models registered moderate performance across multiple datasets. Concretely, XGBoost demonstrated superior performance in the measures of accuracy and F1 score, while the random forest algorithm excelled in precision. Similarly, LightGBM outperformed the other models in recall. It is important to highlight that each of these three models resonates with distinct data normalization techniques: Log scaling aligns with the random forest, MinMax scaling is fitting for LightGBM, and Z-score scaling is most suitable for XGBoost. Additionally, this paper also calculates the average values of differences caused by different datasets for each model to assess the comprehensive performance of the models. The average values of each evaluation metric for the models are shown in Table 8.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Variable** & **Coefficient** & **T-Value** \\ \hline Age & 0.000171 & 0.06 \\ Pla & \(-\)0.0022124 & \(-\)1.58 \\ Nnr & 0.8228015*** & 8.14 \\ Frn & 0.01933*** & 11.88 \\ Lcr & 0.0134422*** & 8.86 \\ Constant & \(-\)0.0126499 & \(-\)1.52 \\ \hline \hline \multicolumn{3}{l}{_N:_ 1972} \\ \multicolumn{3}{l}{_R-Square:_ 0.1333} \\ \multicolumn{3}{l}{_Adj R-Square:_ 0.1311} \\ \hline \hline \end{tabular} Note: *** \(p<0.01\).  
\end{table}
Table 6: Regression results of robustness test.  
The performance analysis suggests stable and robust evaluation by the random forest algorithm across various datasets, outperforming the XGBoost algorithm. Hence, irrespective of the dataset standardization methodology employed, the random forest algorithm proves a reliable solution for default prediction.  
This study utilizes the XGBoost algorithm in conjunction with the multi:softmax method, which does not support the return of predicted probabilities, precluding the potential to plot the ROC curve for optimal model performance, as illustrated in Figure 5. Based on the curves' contour, the random forest and LightGBM algorithms demonstrate superior performance, evidenced by their ROC curves' proximity to the top-left corner.  
\begin{table}
\begin{tabular}{c c c c c c} \hline
**Model** & **Standard** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline \multirow{3}{*}{Logistic} & Log & 75.68\% & 77.29\% & 69.37\% & 73.45\% \\  & MinMax & 68.58\% & 70.43\% & 59.73\% & 63.21\% \\  & Z-score & 71.62\% & 72.45\% & 62.67\% & 67.75\% \\ \hline \multirow{3}{*}{SVM} & Log & 76.96\% & 76.48\% & 70.75\% & 75.74\% \\  & MinMax & 71.14\% & 70.38\% & 61.16\% & 67.33\% \\  & Z-score & 68.35\% & 66.16\% & 57.80\% & 63.90\% \\ \hline \multirow{3}{*}{XGBoost} & Log & 80.41\% & 80.61\% & 73.94\% & 79.18\% \\  & MinMax & 79.05\% & 80.28\% & 71.22\% & 77.07\% \\  & Z-score & **81.42\%** & 82.70\% & 74.89\% & **80.00\%** \\ \hline \multirow{3}{*}{DT} & Log & 79.29\% & 78.74\% & 73.97\% & 78.62\% \\  & MinMax & 77.03\% & 76.98\% & 73.91\% & 76.35\% \\  & Z-score & 78.72\% & 78.47\% & 73.97\% & 78.06\% \\ \hline \multirow{3}{*}{RF} & Log & 81.08\% & **82.77\%** & 76.41\% & 79.90\% \\  & MinMax & 79.73\% & 80.91\% & 75.14\% & 78.55\% \\  & Z-score & 80.81\% & 80.89\% & 73.92\% & 79.59\% \\ \hline \multirow{3}{*}{NN} & Log & 71.21\% & 72.36\% & 69.13\% & 71.64\% \\  & MinMax & 72.22\% & 70.70\% & 63.98\% & 70.34\% \\  & Z-score & 72.22\% & 69.08\% & 62.41\% & 69.32\% \\ \hline \multirow{3}{*}{LightGBM} & Log & 80.07\% & 81.13\% & 72.46\% & 78.32\% \\  & MinMax & 81.08\% & 82.56\% & **75.25\%** & 79.72\% \\ \cline{1-1}  & Z-score & 76.35\% & 76.92\% & 68.99\% & 74.26\% \\ \hline \end{tabular}
\end{table}
Table 7: Prediction results of different machine learning models.  
Figure 5: _Cont._