## 3 Methodology Description
### Long Short-Term Modeling
#### 3.2.2 Operation of LSTM  
The first step in the LSTM model is to determine which piece of information to discard from the cell state (CS), considering the current input vector, x(t), and the previous short-term state vector, h(t - 1). The first step is accomplished using a sigmoid layer, which is called the \"forget gate layer\". If the output of the sigmoid as an active function is 0, all previous CS (t - 1) values approaching 0 do not affect future results. However, if the output of the sigmoid as an active function is 1, the previous CS (t - 1) value is completely remembered in order to affect the next step for CS (t) and is given by the following equation [11; 13; 47; 48]:  
\[\mathrm{fg(t)} = \sigma\cdot\bigg{(}\mathrm{w}_{\mathrm{fg}}^{\mathrm{x}}\cdot \mathrm{x}_{t}+\mathrm{w}_{\mathrm{fg}}^{\mathrm{h}}\cdot\mathrm{h}_{\mathrm{t -1}}+\mathrm{b}_{\mathrm{f}}\bigg{)}, \tag{3}\]  
In (3), \(\mathrm{w}_{\mathrm{fg}}^{\mathrm{x}}\) and \(\mathrm{w}_{\mathrm{fg}}^{\mathrm{h}}\) are the weighting vectors for the forget layer that are connected to the input vector denoted as x(t) and the previous short-term state h(t - 1), respectively. In addition, \(\mathrm{b}_{\mathrm{f}}\) is defined as the bias for the forget layers.  
The next step determines the new information to be stored in the CS, which is classified into two subprocesses. First, a sigmoid layer called the input gate layer, ip(t), determines which values to update. Next, a tanh layer creates a vector of new candidate values, Gi(t), that could be added to the state. In the next step, we combine these two steps to update the state ig(t), as shown in (4) to (6) [11; 13; 47; 48]:  
\[\mathrm{ip(t)} = \sigma\bigg{(}\mathrm{w}_{\mathrm{ip}}^{\mathrm{x}}\cdot \mathrm{x}_{t}+\mathrm{w}_{\mathrm{hi}}^{\mathrm{h}}\cdot\mathrm{h}_{\mathrm{t -1}}+\mathrm{b}_{\mathrm{i}}\bigg{)} \tag{4}\] \[\mathrm{Gi(t)} = \mathrm{tanh}\,\mathrm{h}\,\left(\mathrm{w}_{\mathrm{Gi}}^{ \mathrm{x}}\cdot\mathrm{x}_{t}+\mathrm{w}_{\mathrm{Gi}}^{\mathrm{h}}\cdot \mathrm{h}_{\mathrm{t-1}}+\mathrm{b}_{\mathrm{Gi}}\right)\] (5) \[\mathrm{ig(t)} = \mathrm{ip(t)}\,\otimes\mathrm{Gi(t)} \tag{6}\]  
where \(\mathrm{w}_{\mathrm{ip}}^{\mathrm{x}}\) and \(\mathrm{w}_{\mathrm{hi}}^{\mathrm{h}}\) are represented as weighting vectors for the input gate layer that are connected to the input vector denoted as x(t) and the previous short-term state h(t-1), respectively. The terms \(\mathrm{b}_{\mathrm{i}}\) and \(\mathrm{b}_{\mathrm{Gi}}\) are defined as the bias for the input gate layer.  
The final decision step for the optimal CS (t) is based on the previous two steps, which can be readily obtained as follows [11; 13; 47; 48]:  
\[\mathrm{CS(t)} = \mathrm{fg(t)}\,\otimes\,\mathrm{CS(t}\,-\,1)\,+\,\mathrm{ig(t)} \,\otimes\,\mathrm{Gi(t)}. \tag{7}\]  
First, the sigmoid function \(\sigma\) is determined by whether a CS outputs 0 or 1 for the neuron, denoted as O(t) [11; 13; 47; 48]:  
\[\mathrm{O(t)} = \sigma\cdot\Big{(}\mathrm{w}_{\mathrm{O}}^{\mathrm{x}}\cdot \mathrm{x}_{t}+\mathrm{w}_{\mathrm{O}}^{\mathrm{h}}\cdot\mathrm{h}_{\mathrm{t -1}}+\mathrm{b}_{\mathrm{O}}\Big{)} \tag{8}\]  
Figure 1: Structure of long short-term memory (LSTM) model: the repeating module has three different types of gates, namely, the forget gate, fg(t); the input gate, ig(t) (=ip(t) \(\times\) Gi(t)); the output gate, O(t).  
where \(w_{O}^{x}\) and \(w_{O}^{h}\) can be expressed as weighting vectors for the output gate that are connected to the input vector denoted as \(x(t)\) and the previous short-term state \(h(t-1)\), respectively. In addition, \(b_{o}\) is defined as the bias for the output gate layer. Then, we apply the tanh function, which assigns weights to the values that are passed through, determining the level of importance, ranging from \(-1\) to \(1\) of the CSs, and multiplying it by the output of the sigmoid gate denoted as \(h(t)\)[11; 13; 47; 48]:  
\[h(t) = \;O(t)\;\otimes\;\tanh\;(CS(t)) \tag{9}\]