# A Multi-Head LSTM Architecture for Bankruptcy Prediction with Time Series Accounting Data
## 8 LSTM Architectures for Bankruptcy Prediction  
According to [43], LSTM performs better than a GRU when the sequence is short, although with a matrix input. For this reason, we chose the LSTM approach for our experiments since we were considering eighteen different time series as input, each with a short sequence length, as determined in the first experiments presented in Section 7. In order to study the application of RNNs to bankruptcy prediction, we evaluated two different architectures:  
* Single-input LSTM: This is the most common approach with RNNs. The input was a matrix with 18 rows (the number of accounting variables) and a number of columns equal to the time window selected for the experiment. Moreover, the LSTM unit was composed of a sequence of units as long as the time window. Finally, a dense layer with a Softmax function was used as an output layer for the final prediction.
* Multi-head LSTM: This is one of the main contributions of our research concerning the current state of the art. To deal with a smaller training set due to the temporal window selection and the class imbalance, we developed several smaller LSTM architectures, one for each accounting variable to be analyzed by the model, named LSTM heads. Each network included a short sequence of units equal to the input sequence length and contributed to the latent representation of the company learned by utilizing the accounting variables. Indeed, th output of the multi-head layer was then concatenated and exploited by a two-layer feed-forward network with a Softmax function in the output layer. This architecture aimed to test whether an attention method based on a latent representation of the company that focused on each time series independently could outperform a classical RNN setting.  
Figure 6 summarizes the main differences between the two architectures. The source code for the multi-head LSTM architecture is publicly available on GitHub ([[https://github.com/sowide/Multi-head_LSTM_for_bankruptcy-prediction](https://github.com/sowide/Multi-head_LSTM_for_bankruptcy-prediction)]([https://github.com/sowide/Multi-head_LSTM_for_bankruptcy-prediction](https://github.com/sowide/Multi-head_LSTM_for_bankruptcy-prediction)), accessed on 1 February 2024).  
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline \multirow{2}{*}{**ML Model**} & \multicolumn{5}{c}{**Average Training Time (s)**} \\ \cline{2-6}  & **WL = 1** & **WL = 2** & **WL = 3** & **WL = 4** & **WL = 5** \\ \hline Support Vector Machine & 0.032 & 0.036 & 0.036 & 0.034 & 0.033 \\ Logistic Regression & 0.018 & 0.022 & 0.023 & 0.024 & 0.026 \\ AdaBoost & 0.826 & 1.246 & 1.560 & 1.769 & 1.994 \\ Random Forest & 0.799 & 0.973 & 1.033 & 1.034 & 1.059 \\ Gradient Boosting & 1.022 & 1.833 & 2.545 & 3.097 & 3.555 \\ XGBost & 0.421 & 0.422 & 0.493 & 0.478 & 0.483 \\ CatBoost & 6.670 & 7.059 & 7.088 & 7.383 & 7.398 \\ LightGBM & 0.187 & 0.195 & 0.191 & 0.184 & 0.179 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Average training time (seconds) on the validation set depending on the window length (WL). Times in seconds refer to the average training time for a single run.