# A Multi-Head LSTM Architecture for Bankruptcy Prediction with Time Series Accounting Data
## 10 Final Analysis on the Test Set
### Further Analysis on the Test Set  
Understanding the distribution and the relationships within the feature space is crucial for gaining insights into the behavior of a model. Among the methodologies employed for representing high-dimensional data, one of the most powerful approaches is t-distributed stochastic neighbor embedding (T-SNE) [13]. T-SNE preserves the non-linear structure of data points and tends to maintain the relative distances between neighboring points, which can reveal clusters and patterns. The analysis reported in this subsection aimed to study the distribution of both healthy and bankrupt firms in a reduced-dimensional space, focusing on the capabilities of multi-head and single-input LSTM in identifying an optimal decision boundary. We first leveraged T-SNE for the original input to visualize the data in a two-dimensional space while preserving the pairwise similarities between data points. Each data point represented a firm, allowing us to gain an intuitive understanding of its inherent structures. The results are shown in Figure 9(a). We performed an analysis using the best window lengths identified on the validation set (WL = 4 and WL = 5). We could infer that bankruptcy-prone firms usually did not cluster in a specific region of the feature space of the input. This observation highlights how challenging it is to classify bankrupt companies because there is some degree of overlap with healthy ones. Following the outcomes achieved from the T-SNE analysis of the original test set, we focused on how the LSTM networks represented each firm in their latent space before classifying them, as depicted in Figure 10. These snapshots offer some insights into the decision boundaries identified by the two recurrent architectures. The decision boundaries for LSTM are depicted in Figure 9(b), and those for multi-head networks in Figure 9(c). \begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **LSTM** & **Multi-Head LSTM** & **LSTM** & **Multi-Head LSTM** \\ \hline TP & **88** & 75 & **89** & 71 \\ TN & 1233 & **2158** & 1071 & **2085** \\ FN & **8** & 21 & 2 & 20 \\ FP & 1591 & 666 & 1573 & **559** \\ AUC score & 0.832 & **0.847** & 0.802 & **0.825** \\ BAC & 0.677 & **0.773** & 0.692 & **0.784** \\ Micro F1 & 0.772 & **0.797** & 0.777 & **0.793** \\ Macro F1 & 0.53 & **0.55** & 0.528 & **0.542** \\ Type I error & 21.74 & **18.45** & 22.65 & **20.5** \\ Type II error & **23.95** & 28.125 & 21.97 & 21.97 \\ Rec bankruptcy & **0.76** & 0.71 & 0.78 & 0.78 \\ Pr bankruptcy & 0.106 & **0.117** & 0.106 & **0.115** \\ Rec healthy & 0.782 & **0.815** & 0.773 & **0.795** \\ Pr healthy & **0.989** & 0.988 & 0.99 & 0.99 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Overall results on the test set (2015–2018) with single-input LSTM and multi-head LSTM. Rec indicates recall and Pr indicates precision. The best results were achieved for WL = 4 by multi-head LSTM. Our analysis revealed that these embeddings effectively distributed firms, with bankrupt firms forming distinct clusters. This demonstrates the models' ability to capture meaningful patterns in the data, particularly concerning bankruptcies. The models consistently demonstrated the ability, whether the window length was equal to 4 or 5, to discriminate between healthy and bankrupt firms in a two-dimensional environment. The models' constant ability to identify significant patterns linked to financial distress showed their robustness. By comparing the latent representations achieved by single-input LSTM with those achieved by the multi-head model, it was evident that the multi-head model achieved a more scattered representation that led to smaller overlaps between healthy and bankrupt firms. This result is also in line with the results presented in Table 6, checking the number of false positives (FPs) achieved by the two models. Thanks to this, multi-head LSTM outperformed single-input LSTM in terms of AUC, BAC, and FP for both window lengths.