# DERN: Deep Ensemble Learning Model for Short- and Long-Term Prediction of Baltic Dry Index
## 3 Method
### Deep Ensemble Recurrent Network  
The design system of our approach is depicted in Figure 2, and the training process is expressed in Algorithm 1. Firstly, the pre-processed data is independently trained by Deep RNN, LSTM, and GRU. After those models converge in learning BDI data, in the testing phase, each of the predictions of RNN, LSTM, and GRU, represented as \(P_{1}\), \(P_{2}\) and \(P_{3}\), are summarized using weighting \(w_{p1}\), \(w_{p2}\) and \(w_{p3}\), respectively, to predict the \(P_{f}\) that represents the next value of the BDI. The values of \(w_{p1}\), \(w_{p2}\) and \(w_{p3}\) are learned in supervised learning using basic forward- and back-propagation techniques as in the neural network. As explained in Section 2.2, all of them are powerful sequence models. Deep RNN is the best recurrent model, which is able to learn time dependency to predict the next value; however, it cannot be combined with long-term dependency. LSTM is equipped with complex cell memory to handle long-term dependency. As for GRU, whereas it harnesses cell memory as well, it uses a simpler cell than LSTM. Therefore, for obtainment of more accurate results, we combined all of those models in an ensemble called deep ensemble recurrent network (DERN). The purpose behind this was to minimize the error rate, since in terms of memory cells, RNN is the simplest (under-fit) model, while GRU is a simpler model than LSTM albeit more complex (over-fit) than RNN, which complexity can be denoted as RNN \(<\) GRU \(<\) LSTM. Moreover, in machine-learning theory, there is no method that is universally better than any other method (the \"no free lunch\" theorem), and each method may make mistakes in different facets of operation. Stacking of multiple different sequential models may lead to performance improvement over individual models. The multi-model ensemble is a technique by which the predictions of the collection of models are given as inputs to a second-stage learning model. The second-stage model is trained to combine the predictions from the first-stage models optimally in order to obtain a final set of predictions. The mechanism that we used to make short- and long-term predictions in the present study is depicted in Figure 3. Figure 3a is one-step-ahead (short-term) prediction model. Therefore, we transform the univariate time-series data of BDI into \(x_{i}\) as a predicted variable and \(x_{i+1}\) as a response variable. Afterwards, to predict the \(x_{i+2}\), \(x_{i+1}\) is appended to the training data. In Figure 3b meanwhile, the model predicts multi-values of BDI at a time; thus, it predicts multiple sequences from \(x_{i+1}\) to \(x_{n}\) at once, where n is the number of step predictions. The challenging part of this technique is creating a model that performs long-sequence prediction at once. Figure 2: Ensemble deep RNN, LSTM and GRU model. ```
Input :\(D=\left\{x_{i},y_{i}\right\}_{j=1}^{m}\) output :\(P_{f}\) for\(t\gets 1\) to \(T\)do  Learn \(M_{1}\) based on \(D\) // \(M_{1}\) is RNN model with its hyper parameter  end for for\(t\gets 1\) to \(T\)do  Learn \(M_{2}\) based on \(D\) // \(M_{2}\) is LSTM model with its hyper parameter  end for for\(t\gets 1\) to \(T\)do  Learn \(M_{3}\) based on \(D\) // \(M_{3}\) is GRU model with its hyper parameter  end for for\(t\gets 1\) to \(m\)do \(P_{1}=D(M_{1})\) // Predict \(P_{1}\) given data \(D\) and model \(M_{1}\) \(P_{2}=D(M_{2})\) // Predict \(P_{2}\) given data \(D\) and model \(M_{2}\) \(P_{3}=D(M_{3})\) // Predict \(P_{3}\) given data \(D\) and model \(M_{3}\)  end for for\(t\gets 1\) to \(T\)do  Learn \(w_{P1}\), \(w_{P2}\), \(w_{P3}\) based on \(D\) // \(w_{P1}\), \(w_{P2}\), \(w_{P3}\) is weight of \(P_{1}\), \(P_{2}\) and \(P_{3}\), respectively  end for for\(t\gets 1\) to \(m\)do \(P_{f}=P_{1}(w_{P1})+P_{2}(w_{P2})+P_{3}(w_{P3})\)  end for
```  
**Algorithm 1**DERN  
We conducted some extensive experiments to decide the hyper-parameters of our models.