# A Multi-Head LSTM Architecture for Bankruptcy Prediction with Time Series Accounting Data
## 3 Recurrent Neural Networks  
A Recurrent Neural Network (RNN) is a deep learning architecture that aims to process sequences of values in the form \(x^{(1)}\), \(x^{(2)}\)...., \(x^{(t)}\). This ability is due to the network's parameter sharing across different parts of a model, which makes it possible to extend and apply the model to examples of different forms. Moreover, parameter sharing also allows the model to preserve generalization across the sequence since the same parameters (weights) are used for each value of the time index, while a traditional fully connected feed-forward network would have separate parameters for each input feature. The time index refers to the position in the sequence. A Recurrent Neural Network is generally composed of a single unit of processing that produces an output \(y\) at each time step and has recurrent connections in general from the hidden units \(a^{<l>}\) and optionally from the output. When an RNN has only recurrent connections from the hidden units, it processes an entire sequence and then produces a single output. Figure 1 shows a generic RNN structure that processes a sequence of \(t\) elements with recurrent connections only from the hidden units. Input, output, and recurrent hidden states are propagated using different weight matrices whose elements are learned during training via the back-propagation through time algorithm [33]. Equations (1) and (2) describe the internal behavior of an RNN unit. The initial hidden state \(a^{0}\) is generally equal to zero. In general, for a generic time index \(j\), the hidden state is computed as a weighted sum of the previous state \(a^{j-1}\) and the current input \(x^{j}\) plus a bias term. After that, an activation function \(\sigma\) is applied to the result as in fully connected networks. The output at each time step only depends on the current internal state plus a different bias \(b_{y}\). The two activation functions to estimate the hidden state and the output may differ. Additionally, the weight matrices \(W_{aa}\), \(W_{xa}\), and \(W_{ya}\) play crucial roles in the computations:  
* \(W_{aa}\): Weight matrix from the previous hidden state \(a^{<j-1>}\) to the current hidden state \(a^{<l>}\). * \(W_{xa}\): Weight from the input at time \(j\) (\(x^{<j>}\)) to the current hidden state \(a^{<j>}\). * \(W_{ya}\): Weight matrix from the hidden state \(a^{<j>}\) to the output \(y^{<l>}\). \[a^{<j>} =\sigma(W_{aa}\cdot a^{<j-1>}+W_{xa}\cdot x^{<j>}+b_{a})\] (1) \[y^{<j>} =\sigma(W_{ya}\cdot a^{<j>}+b_{y})\] (2)  
In this way, Recurrent Neural Networks can process entire sequences and can use contextual information when mapping inputs into outputs. Unfortunately, for standard RNN architectures, the range of context that can be in practice is quite limited, especially for long sequences and for more than one sequence in the input (matrix input). The major issue is that the influence of a given input on the hidden state, and therefore on the network output, either decays or blows up exponentially as it cycles around the network's recurrent connections. This effect is often referred to in the literature as the vanishing gradient problem [34]. Several approaches have been presented to solve this issue like the Long-Short Term Memory (LSTM) architecture [35] and the Gated Recurrent Unit (GRU) [36]. In both architectures, several additional components (called gates) are introduced inside the unit to extend the memory of the network in case of a long sequence so that the first part of each  
Figure 1: (**Left**) A generic structure of an unrolled Recurrent Neural Network with recurrent connections from the hidden layer. (**Right**) The resulting RNN when presented as a single unit with recurrent connections. sequence is not forgotten when producing the output (long-term dependency problem) and to prevent the gradient from vanishing. We describe LSTM since it is the architecture used in this research work and because the GRU unit can be taken back to a particular case of the LSTM unit.