# DERN: Deep Ensemble Learning Model for Short- and Long-Term Prediction of Baltic Dry Index
## 2 Background
### Sequential Model  
Recurrent Neural Network (RNN) is a type of neural network with loops that allow for retention of information from the past. Specifically, the loops enable RNN to use information from past time slices to produce output for the current time slice \(t\). Thus, we can say that the decision made at a time slice \(t-1\) affects the decision to be made at \(t\). Therefore, the response of the network to the new data depends on the current input as well as the output from the recent past data. The RNN output calculation is based on iterative calculation of the output of the following two equations:  
\[h_{t}=H(W_{xh}x_{t}+W_{hh}h_{t-1}+b_{h}) \tag{1}\]  
\[y_{t}=w_{hy}h_{t}+b_{y} \tag{2}\]  
In Equations (1) and (2), \(x_{t}\) is the input sequence, \(y_{t}\) is the output sequence, and \(h_{t}\) represents the hidden vector sequence at time slice \(t\) (\(t\) = 1, 2,..., \(T\)); \(W\) and \(b\) represent weight matrices and biases, respectively; and lastly, \(H\) is an activation function used for the hidden layer. The back-propagation through time (BPTT) technique is usually used to train RNNs [23]. However, it is difficult to use BPTT to train traditional RNNs, due to the gradient-vanishing and exploding problem [24]. Errors from later time steps are difficult to propagate back to previous time steps for proper updates of network parameters. To address this problem, the long short-term memory (LSTM) unit has been developed [25]. LSTM is a special type of recurrent neural network with memory cells. These memory cells are the essential component for handling of long-term temporal dependencies in the data. LSTM has the option to add or delete information from this cell state. This operation is done by special structures in LSTM, which are called gates. The three types of gate are _input gate_ (\(i_{t}\)), _forget gate_ (\(f_{t}\)), and _output gate_ (\(o_{t}\)), shown in Equations (3) to (8). \(\bar{C_{t}}\) is a \"candidate\" hidden state that is computed based on the current input and the previous hidden state. \(\bar{C_{t}}\) is the internal memory of the unit. It is a combination of the previous memory, as multiplied by the forget gate, and the newly computed hidden state, as multiplied by the input gate. \(h_{t}\) is the output hidden state, and is computed by multiplying the memory by the output gate [26]. \[h_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i}) \tag{3}\]  
\[h_{t}=\sigma(W_{f}\cdot[h_{t-1},x_{t}]+b_{f}) \tag{4}\]  
\[o_{t}=\sigma(W_{o}\cdot[h_{t-1},x_{t}]+b_{o}) \tag{5}\]  
\[\bar{C_{t}}=\tanh(W_{c}\cdot[h_{t-1},x_{t}]+b_{c}) \tag{6}\]  
\[C_{t}=f_{t}*C_{t-1}+i_{t}*\bar{C_{t}} \tag{7}\]  
\[h_{t}=o_{t}*\tanh(C_{t}) \tag{8}\]  
GRU is another type of RNN with memory cells [27]. It is similar to LSTM but with a simpler cell architecture. GRU also has a gating mechanism to control the flow of information through the cell state, but has fewer parameters and does not contain an output gate. It consists of two gates, \(r\) being a reset gate, and \(z\) an update gate. The reset gate regulates the flow of new input to the previous memory, and the update gate determines how much of the previous memory to keep. The following equations are used in the GRU output calculations:  
\[z_{t}=\sigma(W_{xz}x_{t}+W_{hz}h_{t-1}+b_{z}) \tag{9}\]  
\[r_{t}=\sigma(W_{xx}x_{t}+W_{hr}h_{t-1}+b_{r}) \tag{10}\]  
\[y_{t}=w_{hy}h_{t}+b_{y} \tag{2}\]0  
\[y_{t}=w_{hy}h_{t}+b_{y} \tag{2}\]1  
In previous studies [28; 29], it has been noted that GRU is comparable to, or even outperforms, the LSTM. Regarding the obtainment of high accuracy in prediction of the BDL, in this study, we combined RNN, LSTM, and GRU into an ensemble method. The idea was to combine the predictions from multiple different sequential models. Each model has different strength and weakness, meaning that its predictions are better than any other in a certain condition. Importantly, the models must be good in different ways: they must make different prediction errors.