# Machine Learning for Bankruptcy Prediction in the American Stock Market: Dataset and Benchmarks
## 6 Task TI: Default Prediction with Historical Time Series
### Models Comparison and Selection for Default Prediction  
All the models were trained using the same training set (1999-2011) and compared using the same validation set (2012-2014). However, learning from an imbalanced training set led to unsatisfactory results for the bankruptcy class in the validation set. For this reason, we decided to use a random balanced training set that is evaluated for different runs. Indeed, every model is evaluated over 100 independent and different runs: for every run, the training set is balanced with exactly the same number of bankruptcy examples and a random choice of healthy examples from the same period. The number of features changes according to the number of available variables for the temporal window length selected. A binary classification task is implemented in each model, where the positive class (1) represents a bankruptcy case in the next year, and the negative class (0) represents a healthy case. For RF, AB, GB, and XGB, we used 500 estimators for a fair comparison, while the other specific parameters are the default ones provided in the Scikit-Learn implementations. The ANN is a multi-layer perceptron with three layers. In the first layer, the number of neurons is equal to the number of input features. The hidden layer has half as many neurons as the first layer, and finally, the output layer uses a Sigmoid function to produce a binary prediction. All neurons except for those in the output layer use the Rectified Linear Unit activation (RELU) function. Each Neural Network is trained over 1000 epochs using the early-stopping regularization technique that prevents overfitting when the validation loss tends to increase for a patience number of times. We compared all the models using the average Area Under the Curve (_AUC_) over 100 runs. Figure 5 reports the results we achieved trying to predict bankruptcy in the most common setting that exploits only the accounting variables of the last available fiscal year. On average, the ensemble tree-based ML models perform better than the others except for Adaboost. The best average result is achieved with the Random Forest (\(AUC=0.748\)), although the Neural Network achieves the best \(AUC\) in a single run (\(AUC=0.856\)). However, the ANN exhibits the highest variability in the results, which led to a lower average AUC (\(AUC=0.653\)). Figure 6 reports the comparison among the models when using more than one year from the prediction (WL from two to five years). As a result of this experiment, we can assert that on average increasing the number of fiscal years in the input does not seem to impact the average AUC significantly:  
* For WL = 2, best average AUC = 0.75 with Random Forest
* For WL = 3, best average AUC = 0.72 with Random Forest and Gradient Boosting
* For WL = 4, best average AUC = 0.74 with XGBoost
* For WL = 5, best average AUC = 0.754 with Random Forest  
The highest average value is reached by Random Forest using a window length of 5 years. However, as in the first case, in general, the ensemble models exhibit better performance. However, the best absolute result is reached by the Neural Network again with \(AUC=0.85\). It should be highlighted that the Neural Network's high variability  
Figure 4: Bankruptcy and healthy sample distribution in the training (2000–2011), validation (2012–2014), and test (2015–2018) sets by varying the length of the window (WL) considered. concerning the results can be imputed to the random weights' initialization for every run. Probably performing several runs over the same training set could reduce this problem. Finally, Table 3 shows the variance for each model considering every window length. Figure 5: The box plot shows the locality, spread, and skewness groups of AUC values through their quartiles achieved for each model in 100 different runs with a different balance training set. The reported AUC values (horizontal axis) are related to the case of WL = 1 where all models use only variables from the last year.