# Ensemble Deep Learning Models for Forecasting Cryptocurrency Time-Series
## 3 Advanced Deep Learning Techniques  
### Long Short-Term Memory Neural Networks  
Long Short-term memory (LSTM) [19] constitutes a special case of recurrent neural networks which were originally proposed to model both short-term and long-term dependencies [20; 21; 22]. The major novelty unit in a LSTM network is the memory block in the recurrent hidden layer which contains memory cells with self-connections memorizing the temporal state and adaptive gate units for controlling the information flow in the block. With the treatment of the hidden layer as a memory unit, LSTM can cope the correlation within time-series in both short and long term [23].  
More analytically, the structure of the memory cell \(c_{t}\) is composed by three gates: the input gate, the forget gate and the output gate. At every time step \(t\), the input gate \(i_{t}\) determines which information is added to the cell state \(S_{t}\) (memory), the forget gate \(f_{t}\) determines which information is thrown away from the cell state through the decision by a transformation function in the forget gate layer; while the output gate \(o_{t}\) determines which information from the cell state will be used as output.  
With the utilization of gates in each cell, data can be filtered, discarded or added. In this way, LSTM networks are capable of identifying both short and long term correlation features within time series. Additionally, it is worth mentioning that a significant advantage of the utilization of memory cells and adaptive gates which control information flow is that the vanishing gradient problem can be considerably addressed, which is crucial for the generalization performance of the network [20].  
The simplest way to increase the depth and the capacity of LSTM networks is to stack LSTM layers together, in which the output of the \((L-1)\)th LSTM layer at time \(t\) is treated as input of the \(L\)th layer. Notice that this input-output connections are the only connections between the LSTM layers of the network. Based on the above formulation, the structure of the stacked LSTM can be describe as follows: Let \(h_{t}^{L}\) and \(h_{t}^{L-1}\) denote outputs in the \(L\)th and \((L-1)\)th layer, respectively. Each layer \(L\) produces a hidden state \(h_{t}^{L}\) based on the current output of the previous layer \(h_{t}^{L-1}\) and time \(h_{t-1}^{L}\). More specifically, the forget gate \(f_{t}^{L}\) of the \(L\) layer calculates the input for cell state \(c_{t-1}^{L}\) by  
\[f_{t}=\sigma\left(W_{f}^{L}[h_{t-1}^{L},h_{t}^{L-1}]+b_{f}^{L}\right)\,,\]  
where \(\sigma(\cdot)\) is a sigmoid function while \(W_{f}^{L}\) and \(b_{f}\) are the weights matrix and bias vector of layer \(L\) regarding the forget gate, respectively. Subsequently, the input gate \(i_{t}^{L}\) of the \(L\) layer computes the values to be added to the memory cell \(c_{t}^{L}\) by  
\[i_{t}^{L}=\sigma\left(W_{i}^{L}[h_{t-1}^{L},h_{t}^{L-1}]+b_{i}^{L}\right)\,,\]  
where \(W_{i}^{L}\) is the weights matrix of layer \(L\) regarding the input gate. Then, the output gate \(o_{t}^{L}\) of the \(L\)th layer filter the information and calculated the output value by  
\[o_{t}^{L}=\sigma\left(W_{o}^{L}[h_{t-1}^{L},h_{t}^{L-1}]+b_{o}^{L}\right)\,,\]  
where \(W_{f}^{o}\) and \(b_{o}\) are the weights matrix and bias vector of the output gate in the \(L\) layer, respectively. Finally, the output of the memory cell is computed by  
\[h_{t}=o_{t}^{L}\cdot\tanh\left(c_{t}^{L}\right)\,,\]  
where \(\cdot\) denotes the pointwise vector multiplication, tanh the hyperbolic tangent function and \[c_{t}^{L} = f_{t}^{L}\cdot c_{t-1}^{L}+i_{t}^{L}\cdot\tilde{c}_{t-1}^{L},\] \[\tilde{c}_{t}^{L} = \tanh\left(W_{\tilde{c}}^{L}[h_{t-1}^{L},h_{t}^{L-1}]+b_{\tilde{c} }^{L}\right).\]