## 3 Proposed Methods
### Recurrent Networks Classifiers
#### 3.2.2 Long Short-Term Memory  
Long short-term memory (LSTM) is an enhanced RNN type with memory cells [55]. These memory cells are introduced for handling long-term temporal dependencies in the data. It makes it possible to add or remove information from the current cell state. The input gate (\(\mathbf{i}_{t}\)), the forget gate (\(\mathbf{f}_{t}\)), and the output gate (\(\mathbf{o}_{t}\)) is computed to control this memory. Thus, an important feature that appeared early in the input sequence can be propagated over a long distance by LSTM units and capture potential long-distance dependencies. Unlike RNN, LSTM has three logistic sigmoid gates and one tanh layer. The gates limit the information that passes through the cell. They determine which information is needed by the following cell and which can be rejected. The output value is typically in the range of 0 to 1, where 0 indicates \"reject all\" and 1 indicates \"include all\". The architecture of the LSTM unit is described in Figure 6, and the formulas to compute the time-step \(t\) for each hidden state in the LSTM unit are explained in the following equations [56]:  
\[\mathbf{i}_{t}\;=\;\sigma(\mathbf{W}_{i}\cdot[\mathbf{h}_{t-1},\mathbf{x}_{t}]\;+\;\mathbf{b}_{i}) \tag{3}\]  
\[\mathbf{f}_{t}\;=\;\sigma\Big{(}\mathbf{W}_{f}\cdot[\mathbf{h}_{t-1},\mathbf{x}_{t}]\;+\;\mathbf{b} _{f}\Big{)} \tag{4}\]  
\[\mathbf{o}_{t}\;=\;\sigma(\mathbf{W}_{o}\cdot[\mathbf{h}_{t-1},\mathbf{x}_{t}]\;+\;\mathbf{b}_{o}) \tag{5}\]  
\[\widetilde{\mathbf{C}}_{t}\;=\;\tanh(\mathbf{W}_{c}\cdot[\mathbf{h}_{t-1},\mathbf{x}_{t}]\;+\; \mathbf{b}_{c}) \tag{6}\]  
\[\mathbf{C}_{t}\;=\;\mathbf{f}_{t}\;*\;\mathbf{C}_{t-1}\;+\;\mathbf{i}_{t}\;*\;\widetilde{\mathbf{C }}_{t} \tag{7}\]  
\[\mathbf{h}_{t}\;=\;\mathbf{o}_{t}\;*\;\tanh(\mathbf{C}_{t}) \tag{8}\]  
Each LSTM cell takes three inputs: \(\mathbf{h}_{t-1}\), \(\mathbf{C}_{t-1}\), and input \(\mathbf{x}_{t}\), as well as produces two outputs: \(\mathbf{h}_{t}\) and \(\mathbf{C}_{t}\). \(\mathbf{h}_{t}\) is the hidden state, \(\mathbf{C}_{t}\) is the cell state or memory, and \(\mathbf{x}_{t}\) is the current data point or input for a given time \(t\). The first sigmoid layer (\(\sigma\)) contains two inputs, namely \(\mathbf{h}_{t-1}\) and \(\mathbf{x}_{t}\), where \(\mathbf{h}_{t-1}\) is the previous cell's hidden state. It is referred to as the forget gate (\(\mathbf{f}_{t}\) ) since its output controls the amount of the previous cell's information to be included. The output is a number in the range [0, 1] which is multiplied (point-wise) by the previous cell state \(\mathbf{C}_{t-1}\). The second sigmoid layer is the input gate (\(\mathbf{i}_{t}\)), which determines what new information to add to the cell. The tanh layer then creates candidate vector \(\widetilde{\mathbf{C}}_{t}\). These two layers evaluate the information to be stored in the cell state by the point-wise multiplication of ( \(\mathbf{i}_{t}\;*\;\widetilde{\mathbf{C}}_{t}\) ). The result is then added to the forget gate's prior cell state (\(\mathbf{f}_{t}\;*\;\mathbf{C}_{t-1}\)) to generate the current cell state \(\mathbf{C}_{t}\). Next, the output of the cell is computed using a sigmoid and tanh layer. The sigmoid layer determines which cell state will be output, while the tanh layer adjusts the output in the range of [\(-1,1\)]. Finally, a point-wise multiplication of these two layers produces the cell's output \(\mathbf{h}_{t}\).  
Figure 5: RNN architecture.