# Machine Learning for Bankruptcy Prediction in the American Stock Market: Dataset and Benchmarks
## 5 Metrics for Imbalanced Bankruptcy Prediction Tasks  
The two bankruptcy prediction tasks we propose in the following sections have been implemented as binary prediction tasks where the positive class (1) indicates bankruptcy while the negative class (0) means that a company has been classified as healthy. To compare our models and investigate their effectiveness, we used different metrics considering the imbalance condition in the validation and the test sets. These metrics will be critically discussed in Section 8 in light of our results for the two tasks. According to the binary classification task we used to formulate the bankruptcy prediction problem, the following variables represent:  
* **True positive (TP)**: The number of bankrupted companies that have been correctly predicted as such. * **False negative (FN)**: The number of bankrupted companies that have been wrongly predicted as healthy firms. * **True negative (FN)**: The number of actually healthy companies that have been correctly predicted as such. * **False positive (FP)**: The number of healthy companies that have been wrongly predicted as bankrupted by the model. Since the validation and test sets are both imbalanced for both tasks with a prevalence of healthy companies (\(\sim\)96-97%), we did not compare the models in terms of accuracy of the model. Indeed, the proportion of correct matches would be insufficient in assessing the model performance. We firstly use the Area Under the Curve (AUC) for all the comparisons as it is commonly used in the literature to compare the performance of models on imbalanced datasets and specifically to evaluate the bankruptcy models in general. AUC measures the ability of a classifier to distinguish between classes and is used as a summary of the Receiver Operating Characteristic (ROC) curve. The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. In addition, we investigated other important metrics that can be used to clarify the models' performance depending on the target stakeholders. These are the precision, recall, and \(F_{1}\) scores for each class. The precision achieved for a class is the accuracy of that class' predictions. The recall (sensitivity) is the ratio of the class instances that are correctly detected by the classifier. The \(F_{1}\) score is the harmonic mean of precision and recall: whereas the regular mean treats all values equally, the harmonic one gives more weight to low values. As a consequence, a high \(F_{1}\) score for a certain class is achieved only if both its precision and recall are high. Equations (11)-(13) report how these quantities are computed for the positive class. The definitions for the negative class are exactly the same by inverting positives with negatives. \[Precision=\frac{TP}{(TP+FP)} \tag{9}\]  
\[Recall=\frac{TP}{(TP+FN)} \tag{10}\]  
\[F_{1}score=\frac{2}{\frac{1}{Precision}+\frac{1}{Recall}} \tag{11}\]  
Moreover, we computed and reported two other global metrics for the classifier that have been selected because they enable an overall evaluation of the classifier on both classes:  
* **The macro-\(F_{1}\) score**: The macro-\(F_{1}\) score is computed as the arithmetic mean of the \(F_{1}\) score of all the classes. * **The micro-\(F_{1}\) score**: It is used to assess the quality of multi-label binary problems. It measures the \(F_{1}\) score of the aggregated contributions of all classes. Finally, we used two other metrics that are often evaluated in the bankruptcy prediction models. Because bankruptcy is a rare event, using the classification accuracy to measure a model's performance is misleading since it assumes that _type I error_ (Equation (12)) and _type II error_ (Equation (13)) are equally costly. Actually, the cost of false negatives is much higher than that of false positives for a financial institution. An error has a different cost depending on the class that has been incorrectly predicted.