## 2 Multiple-Criteria Decision-Making
### The TOPSIS Method
It is worth pointing out that the geometric mean is closely linked to the distance to perfect classification in the ROC space:

\[\textit{G-mean}=\sqrt{sensitivity\cdot specificity}. \tag{3}\] \begin{table}
\begin{tabular}{l l l} \hline \hline  & \multicolumn{3}{c}{**Predicted Class**} \\ \hline True class & True-positive (TP) & False-negative (FN) \\  & False-positive (FP) & True-negative (TN) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Confusion matrix for a two-class problem. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **\#Variables** & **\#Positive** & **\#Negative** & **\#Instances** & **IR** \\ \hline Australian & 14 & 307 & 383 & 600 & 0.80 \\ sabiSPQ & 16 & 472 & 472 & 944 & 1.00 \\ Polish & 30 & 128 & 112 & 240 & 1.14 \\ German & 24 & 700 & 300 & 1000 & 2.33 \\ Thomas & 12 & 902 & 323 & 1225 & 2.79 \\ Iranian & 27 & 950 & 50 & 1000 & 19.00 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Overview of the databases used in the experiments. On the other hand, the F-measure is defined as follows:

\[\textit{F-measure}=\frac{2\cdot sensitivity\cdot precision}{precision+sensitivity}, \tag{4}\]

where \(precision=TP/(TP+FP)\). Finally, the root mean squared error (RMSE) corresponds to a standard performance evaluation metric widely-used in a variety of classification problems. Let \(p_{1}\), \(p_{2}\),..., \(p_{m}\) and \(a_{1},a_{2},\ldots,a_{m}\) be the predicted and actual outputs on the test samples, respectively. The root mean squared error allows for measuring the difference between the predicted outputs and the true labels, estimating the deviation of the prediction model from the target value [44]:

\[RMSE=\sqrt{\frac{1}{m}\sum_{i=1}^{m}(p_{i}-a_{i})^{2}}. \tag{5}\]

### Experimental Protocol

As databases are small in size, the performance of the classifiers were evaluated with the 5-fold cross-validation method because this seems to be a trustworthy strategy. Each data set was randomly partitioned into five stratified subsets of equal size: for each round, four blocks were used for training a learning algorithm and the remaining one for testing purposes (see Figure 1). In addition, ten repetitions were run for each trial in order to achieve more stable and reliable outcomes. Finally, the prediction results of all classifiers on the seven criteria were averaged across the 50 runs and then analyzed with the TOPSIS and PROMETHEE methods. ## 4 Results

Tables 5-10 provide the results of each classifier on the seven performance assessment criteria (accuracy, root mean squared error, true-positive and true-negative rates, AUC, geometric mean, and F-measure) for each database. On the other hand, Table 11 reports the mean value across all data sets generated by each prediction model on each metric, which is here used to illustrate the performance of that classifier. For each performance metric, the best performing algorithm has been highlighted in boldface. naive Bayes classifier was the best performing algorithm in terms of TN-rate. Even a more obvious example is for the results over the Thomas database: the Bayesian belief network, logistic regression, MLP, and SVM achieved the highest rates when using the accuracy, the naive Bayes classifier was the model with the highest true-negative rate and geometric mean, and MLP and random forest were the best algorithms on the F-measure. These results show that there was a significant discrepancy regarding the set of criteria. Consequently, different conclusions about the best performing method could be drawn based on the performance assessment metric used. These conflicting outcomes depict a realistic scenario in which a pool of analysts or decision-makers might make very different decisions depending on the criteria used to measure the performance of a credit granting decision system. In our opinion, this reflects an illustrative example of real-life applications where the MCDM techniques should be taken into consideration for making more consistent, trustworthy decisions.