# A Deep Learning Approach for Credit Scoring Using Feature Embedded Transformer
## 4 Experimental Results
### Parameter Analysis  
In this section, we analyzed the influence of different hyper-parameters on the performance of the FE-Transformer model. We selected two important parameters for analysis, the number of heads and the number of transformer layers. As can be seen in Figure 7, the experimental results show that, with an increase in parameters, the performance of the model increases first and then decreases. When the number of heads is set to 4, the KS and AUC of the FE-Transformer achieve the highest values, and when the number of transformer layers is set to 2, the KS and AUC of the FE-Transformer achieve the highest values. The reason for this may be that, when the hyper-parameters value is very small, the model training is not enough, so the performance of the model is general, and when the hyper-parameter values are too large, overfitting problems occur, which affect the performance of the FE-Transformer. Figure 5: Performance comparison of models only using behavioral data. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Train Set**} & \multicolumn{2}{c}{**Test Set**} \\ \hline  & **KS** & **AUC** & **KS** & **AUC** \\ \hline LR & 0.25 & 0.670 & 0.251 & 0.658 \\ XGBoost & 0.262 & 0.679 & 0.26 & 0.665 \\ LSTM & 0.273 & 0.7 & 0.26 & 0.682 \\ AM-LSTM & 0.31 & 0.707 & 0.313 & 0.71 \\ FE-Transformer & 0.33 & 0.731 & 0.32 & 0.72 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Results of models using all data. Figure 6: Performance comparison of models with all data. For the deep learning model containing all data, when we fuse the feature data with the data output from the deep learning model, we added a batch normalization layer. Batch normalization can normalize the data and improve the generalization ability of neural networks [32]. In order to verify the impact of batch normalization on the performance of the model, we conducted a comparative experiment on whether to conduct batch normalization. The results are represented in Table 4 and Figure 8. For the three deep learning models LSTM, AM-LSTM, and FE-Transformer, the performance of the models with batch normalization significantly exceeds that of the models without batch normalization. The reason for this may be that batch normalization can make the output of the deep learning model be consistent with the dimension of the feature data, which is conducive to the use of the gradient descent algorithm to optimize the model. Finally, to analyze the impact of behavioral data on credit scores, we chose the XGBoost model using only the behavioral data for analysis. For this model, the input of the model is the frequency of each event. After building the XGBoost model, we extracted the Top 15 important features. The feature importance score represents the usefulness of the input feature to the user's credit default prediction; the results are shown in Figure 9. In consideration of commercial confidentiality requirements, we desensitized the event names. The results show that the feature importance of different events varies greatly, and some events have a significant prediction effect on user default risk. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**With Normalization**} & \multicolumn{2}{c}{**Without Normalization**} \\ \hline  & **KS** & **AUC** & **KS** & **AUC** \\ \hline LSTM & 0.26 & 0.682 & 0.175 & 0.61 \\ AM-LSTM & 0.313 & 0.71 & 0.21 & 0.6 \\ FE-Transformer & 0.32 & 0.72 & 0.24 & 0.63 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance comparison of deep learning models with normalization and without normalization. Figure 7: Influence of different hyper-parameters on the performance of FE-Transformer model. The FE-Transformer model proposed in this research outputs the predicted user default probability. The probability value is between 0 and 1. Based on this probability value, the user's credit score can be calculated. The credit score is used as the basis for loan approval and pricing.