# A Multi-Head LSTM Architecture for Bankruptcy Prediction with Time Series Accounting Data
## 9 Results
### Statistical Analysis  
The experimental results described in the previous section and presented in Table 5 were computed as an average over 500 different runs and show that our model achieved much better AUCs with respect to the other models (LSTM, Random Forest, and CatBoost). We further analyzed these results to prove that our model's performance was statistically significant. According to [44], a common way to test whether the difference between two classifiers' results over different datasets or runs is non-random is to compute a paired \(t\)-test, which checks whether the difference in their average performance is significantly different from zero.  
However, one of the \(t\)-test's requirements is that the differences between the two random variables compared are distributed normally. As we show in Figure 8, none of the performance differences between each model and our multi-head LSTM architecture were normally distributed; for this reason, according to [45], we used the Wilcoxon Signed-Rank (WSR) test ([46]), a non-parametric alternative to the paired \(t\)-test that ranks the differences in the performance of two classifiers for each run, ignoring the signs, and compares the ranks for the positive and negative differences. We set to 0.05 the \(p\)-value under which we rejected the null hypothesis (the two distributions had the same median, and thus the performance difference could be considered random).  
\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline  & **Avg AUC** & **Max AUC** & **Avg Training Time (s)** & **Avg AUC** & **Max AUC** & **Avg Training Time (s)** \\ \hline Multi-head LSTM & **0.813356** & 0.837 & 103.86 & 0.79 & 0.828 & 82.60 \\ LSTM & 0.8026 & **0.8415** & 51.80 & **0.7928** & **0.864** & 42.86 \\ Random Forest & 0.75 & 0.777 & 1.035 & 0.762 & 0.794 & 1.062 \\ CatBoost & 0.731 & 0.767 & 7.388 & 0.750 & 0.797 & 7.432 \\ \hline \multicolumn{7}{c}{**WL = 4**} & \multicolumn{4}{c}{**WL = 5**} \\ \hline \hline \end{tabular}
\end{table}
Table 5: Average and max AUC achieved on the validation set (2012–2014) by Random Forest, CatBoost, single-input LSTM and multi-head LSTM. The average training time for each model is also reported for WL = 4 and WL = 5.  
Figure 7: The box plot shows the locality, spread, and skewness groups of the AUC values through their quartiles achieved for each model over 500 runs with a different balanced training set. The lines (whiskers) extending from the boxes indicate the variability outside the upper and lower quartiles. The the orange line represents the median between the first and third quartiles. The circular points (fliers) are those past the end of the whiskers.  
In our case, we achieved a \(p\)-value equal to 0 for each comparison with multi-head LSTM over the same 500 runs when models were trained using the same balanced training set, and the AUC was evaluated over the same unbalanced validation set. In light of these results, we can conclude that our model showed better performance in the period between 2012 and 2014. Finally, since the main goal of this analysis is to build a model that can generalize on unseen samples that were not used during the design phase, we report the next section, our final analysis of the performance of the best models on the test set to better prove the benefits of our approach.