# A Deep Learning Approach with Feature Derivation and Selection for Overdue Repayment Forecasting
## 3 Proposed System
### Multi-BiLSTM Layer  
In neural networks such as DNN and CNN, the inputs are independent of each other without temporal dependence, while the recurrent neural network (RNN) considers sequential information in which the data are not only related to the input at this time but also related to the previous input. In other words, the RNN has the ability to memorize. The RNN models the dependency within sequence data extensively, but the accumulation of the gradient product of each time step in backward propagation causes the gradient to disappear when the sequence length is long. On the basis of the RNN, LSTM is a type of neural network with the additional ability of forgetting, which is thus suitable for sequence data and has achieved outstanding results in natural language processing. The shortcoming of the RNN is that there is only one hidden layer state updating inside the network, so the model is relatively simple. All the historical input data are memorized by the RNN without filtering, thus frequently resulting in the long-distance dependence problem. LSTM alleviates the gradient disappearance and explosion problem by introducing several gate units with different functions. LSTM selectively operates on input information which is memorized, forgotten or output to the next layer with a certain weight according to the content importance of the information. All the operations are implemented by multiple computing components called \"cell gates\", including forget gates, input gates and output gates. Firstly, the forget gate calculates the degree of forgetting the historical information, denoted as \(\mathbf{C}\), based on the input data and the hidden state of the previous moment. \[\mathbf{f}_{t}=\sigma(\mathbf{W_{f}}\cdot[\mathbf{h}_{t-1},x_{t}]+\mathbf{b_{f}}) \tag{4}\]  
\(\mathbf{f}_{t}\) will be multiplied by \(\mathbf{C}_{t-1}\) afterwards to forget partial information in \(\mathbf{C}_{t-1}\). The input gate updates the candidate hidden layer value and the hidden layer state. Referring to Equation (5), \(\mathbf{i}_{t}\) determines the proportion of candidate hidden values updating to the hidden value \(\mathbf{C}_{t}\), and the candidate hidden value \(\widehat{\mathbf{C}_{t}}\) is calculated by Equation (6). The hidden layer state \(\mathbf{C}_{t}\) is updated with the forget gate, based on Equation (7). \[\mathbf{i}_{t}=\sigma(\mathbf{W_{i}}\cdot[\mathbf{h}_{t-1},x_{t}]+\mathbf{b_{ i}}) \tag{5}\]  
\[\widehat{\mathbf{C}_{t}}=\tanh(\mathbf{W_{C}}\cdot[\mathbf{h}_{t-1},x_{t}]+ \mathbf{b_{C}}) \tag{6}\]  
\[\mathbf{C_{t}}=\mathbf{f}_{t}\ast\mathbf{C}_{t-1}+\mathbf{i}_{t}\ast\widehat{ \mathbf{C}_{t}} \tag{7}\]The function of the output gate is to output the hidden state in a certain proportion. \[\mathbf{o}_{t}=\sigma(\mathbf{W_{o}}\cdot[\mathbf{h}_{t-1},x_{t}]+ \mathbf{b_{o}}) \tag{8}\] \[\mathbf{h}_{t}=\mathbf{o}_{t}*\tanh(\mathbf{C}_{t}) \tag{9}\]  
Historical information does not flow into the future state entirely, while essential information is preserved and useless information is forgotten after the three cell gates. LSTM and the RNN both process sequence data in one direction, and \(\mathbf{h}_{t}\) is determined only by \(x_{t}\) and \(\mathbf{h}_{t-1}\), ignoring the correlation between future events and current events. BiLSTM complements LSTM to address this problem of using a single direction. By contrast, the input sequence is processed in the reverse order simultaneously to obtain another hidden state, \(\mathbf{h}_{tb}\), thus representing a sequence in two directions, and two hidden states are concatenated afterwards to get the final hidden state \(\mathbf{h}_{t}^{{}^{\prime}}=[\mathbf{h}_{tf};\mathbf{h}_{tb}]\). Consider a sequence of overdue repayment flags of length L: \(\mathbf{X}=\{x^{(1)},x^{(2)},\ \ldots,\ x^{(L)}\}\), where each point \(x^{(t)}\in\{0,1\}\) in the sequence represents whether the \(t\)-th repayment behavior is overdue (when \(x^{(t)}=1\)) or not (when \(x^{(t)}=0\)).