# A Partially Interpretable Adaptive Softmax Regression for Credit Scoring
## 4 Experimental Results
### Machine Learning Baselines and Hyperparameter Setting  
For the PIA-Soft model, we used the same neural network architecture for all datasets. The neural network contains two hidden layers with 32 neurons. For hyper-parameters: learning rate, batch size, and epoch number must be pre-defined to train a model. We set the learning rate to 0.001, epoch number for training to 3000 and use a mini-batch with 32 instances at each iteration. An Early Stopping algorithm is used for finding the optimal epoch number based on given other hyper-parameters. For benchmark models, Logistic regression, which have been the most widely used method for binary classification task [13]. Random Forest classification [14], which is ensemble learning method defined as an aggregation of a multiple decision tree classifiers. AdaBoost classification [15], which is boosting algorithm that focuses on classification problems and aims to combine a set of weak classifiers into a strong one. We use a base estimator as a Decision tree classification. XGBoost classification [16], which is a boosting ensemble algorithm, optimizes the objective of function, size of the tree and the magnitude of the weights are controlled by standard regularization parameters. This method uses Classification and Regression Trees (CART). LightGBM [17] and CatBoost [18] are fast, distributed, high-performance gradient boosting models based on decision tree algorithm, used for classification and many other machine learning tasks. TabNet [19] model is similar to simpler tree-based models while benefiting from high performance, almost identical to deep neural networks. We also use exactly identical architecture to the neural network benchmark with adaptive softmax regression. The hyper-parameters of these baseline classifiers are optimized by random search with 10 cross-validation methods over parameter settings, as shown in Table 2.  
In addition, we apply the most widely used re-sampling techniques with machine learning baselines on the public datasets. The resampling techniques include:  
SMOTE: Synthetic Minority Oversampling Technique, which is the most popular method in this area, generates synthetic samples for the minority class by using k-nearest neighbor (KNN) algorithm [51]. ADASYN: Adaptive Synthetic Sampling [52] uses a weighted distribution for different minority class instances according to their level of difficulty in learning, where more synthetic data is generated for minority class instances that are harder to learn compared to those minority examples that are easier to learn. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **Instances** & **Variables** & **Good/Bad** \\ \hline German & 1000 & 24 & 700/300 \\ Australian & 690 & 14 & 387/307 \\ Taiwan & 6000 & 23 & 3000/3000 \\ FICO & 9871 & 24 & 5136/4735 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of datasets. ROS: Random Over Sampling [53] picks an instance from the minority class instances by using random sampling with replacement until dataset is balanced. Comparison of Predictive Performance  
This empirical evaluation aims to present that our proposed PIA-Soft model could lead to better performance than both the industry-benchmark machine learning models in different evaluation metrics. Table 3 displayed the performance of machine learning models on German dataset to compare them and make a reliable conclusion. For the German dataset (see Table 3), our model indicated the best performance in terms of AUC evaluation metric. Our model achieves 0.798 AUC, 0.781 accuracy, 0.795 f-score, and 0.795 g-mean. The AUC, F-score, and accuracy indicate classifying ability between borrowers as good and bad and g-mean is better at dealing with an imbalanced ratio among credit classes. It is found that with the German dataset, our proposed model shows better predictive performances for AUC evaluation metric, indicating that our model is a suitable approach to the small dataset in credit scoring.