## 3 Methodology
### Word Embedding
#### 3.3.4 Long Short-Term Memory (LSTM)  
where  
\(\text{X}_{t}\) = Input vector at the time t.  
\(\text{h}_{t-1}\) = Previous Hidden state. \(\text{C}_{t-1}\) = Previous Memory state. \(\text{h}_{t}\) = Current Hidden state. \(\text{C}_{t}\) = Current Memory state. \(\text{[x]}\) = Multiplication operation. \(\text{[+]}\) = Addition operation. In this research, we observe the behavior of single-layer, double-layer and triple-layer LSTM with 64 LSTM units and a final layer performing the classification into sentiment class using a dense layer with softmax function. The maximum input is 30 words, and this  
Figure 3: LSTM unit. Figure 2: Dataset word distribution. is denoted as \(X_{t}\), where t is equal to 1... 30. Sentiment analysis is a problem with multiple inputs (n) and a single output. Figure 4 illustrates the LSTM architecture. Long Short-Term Memory is a module in an RNN network that solves missing gradient problems. In general, RNN applies the LSTM network in order to avoid propagation errors. This enables the RNN to learn over many time steps. LSTM contains cells that store information outside of a recurrent network. The cell is like the memory in a computer, deciding when the data needs to be stored, written, read, or deleted by means of the gate, as shown in Figure 4. There are four gates used by LSTM, including: input, forget, output, and new memory container. This study uses four hyperparameters to effectively improve the performance of LSTM. The learning rate is 0.001. The Adam optimizer is utilized, since it has the property of adaptive learning levels [48]. Additionally, 64 LSTM units were used for training this model, although the number of units was dependent on the average length of each review. Finally, based on the previous embedding matrix process, a word vector size of 31,476,300 \(\times\) 300 was used. In this research, we employed three architectures: single-, double- and triple-layer LSTM. Figure 5 shows a single-layer LSTM in which 64 units are arranged in one layer. Several outputs are fed to a dropout layer, followed by a global max pooling layer, a dense layer, and three softmax layers on the top of the network. The expected output is classification as belonging to either the neutral, negative or positive class. The details of the single-layer LSTM architecture are presented in Table 4 below. Figure 6 shows the double-layer LSTM, where the input is an array of embedded words. Each input is accepted by each LSTM unit and followed by a dropout layer. The main difference between this and the single-layer architecture is the additional layer of the LSTM unit before the global max pooling and dense layers. As a result of the thicker architecture, there is an increased number of weighted parameters, leading to a heavier computing load. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer (Type)** & **Output Shape** & **Number of Parameters** \\ \hline Word Embedding & (None, 30, 300) & 31,476,300 \\ \hline LSTM layer & (None, 30, 64) & 93,440 \\ \hline Maxpooling & (None, 64) & 0 \\ \hline Dense layer & (None, 64) & 4160 \\ \hline Dense layer & (None, 3) & 195 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Single-layer LSTM architecture. Figure 4: LSTM architecture. Figure 7 shows the three layers of LSTM, where the input is an array of embedded words. The main difference between the single- and double-layer architectures is the new LSTM layer unit before the global max pooling and dense layers. This architecture is the thickest, with the most trainable weighting parameters. Figure 5: Single-layer LSTM architecture. Figure 6: LSTM double-layer architecture. The details of the double-layer LSTM architecture are presented in Table 5.  
Triple-layer architectures are more complex, with more parameters needing to be trained, and they are expected to produce better accuracy. Table 6 presents the details of the triple-layer LSTM architectures.