## 3 Materials and Methods
### Long Short-Term Memory  
RNNs are a type of feed-forward neural network. RNNs possess a recurrent hidden state activated by using the previous states and can deal with the variable-length sequences and automatically model the contextual information. LSTM is an improved type of RNN (see Figure 3a) designed to solve the exploding/vanishing issues faced by RNNs. The LSTM model contains a chain of recurrent memory units, and each of these chains implicates three \"gates\" with various functions. An LSTM unit contains three gates: input gate \(i_{t}\), forget gate \(f_{t}\), and output gate \(o_{t}\), and memory cell \(c_{t}\) to maintain its state over random time intervals. These gates have been generated to organize the flow of data entering and leaving the memory cell. Suppose \(\tanh\) (.), \(\sigma\)(.), and \(\odot\) are the hyperbolic tangent function, the sigmoid function and product, respectively. \(h_{t}\) is the hidden state vector at time \(t\), and \(x_{t}\) is the input vector at time \(t\). \(W\) and \(U\) illustrate cells for input \(x_{t}\) or the weight matrices  
Figure 2: The flowchart of our proposed model (DNN–MHAT).  
of gates. The hidden state \(h_{t}\) and \(b\) indicate the bias vectors. In the forget gate \(f_{t}\), it defines what information to ignore from the cell state, as indicated by the following equation [41]:  
\[f_{t}=\sigma\Big{(}W_{f}h_{t-1}+U_{f}x_{t}+b_{f}\Big{)} \tag{4}\]  
The input gate \(i_{t}\) defines what must be stored by calculating \(\widetilde{c_{t}}\) and \(i_{t}\) and combining them based on the following equations [41]:  
\[i_{t}=\sigma(W_{i}h_{t-1}+U_{i}x_{t}+b_{i}) \tag{5}\]  
\[\widetilde{c_{t}}=\tanh(W_{c}h_{t-1}+U_{c}x_{t}+b_{c}) \tag{6}\]  
\[c_{t}=f_{t}\ \odot\ t_{t-1}+i_{t}\ \odot+\widetilde{c_{t}} \tag{7}\]  
The output gate \(o_{t}\) defines what information is outputted according to the state of the cell state based on the following equations [41]:  
\[o_{t}=\sigma(W_{o}h_{t-1}+U_{o}x_{t}+b_{o}) \tag{8}\]  
\[h_{t}=o_{t}\ \odot\ \tanh(c_{t}) \tag{9}\]  
The LSTM model is based on serial information, but it is not beneficial, especially if you can reach the following information based on the previous model. Therefore, this is highly useful for sequencing tasks.  
The Bi-LSTM model comprises a forward \(\overrightarrow{h}_{t}\) and a backward \(\overleftarrow{h}_{t}\) LSTM layer (see Figure 3b). The core goal of the Bi-LSTM structure is that the forward layer \(\overrightarrow{h}_{t}\) captures the previous sequential information, and the backward \(\overleftarrow{h}_{t}\) captures the subsequent sequential information; both layers are connected to the same output layer. The most important feature of the BiLSTM architecture is that sequence contextual information is considered. Suppose that the input of time \(t\) is the word embedding \(w_{t}\), at time \(t-1\), the output of the forward layer is \(\overrightarrow{h}_{t-1}\) and the output of the forward hidden layer and the backward hidden layer is \(\overrightarrow{h}_{t-1}\), \(\overleftarrow{h}_{t+1}\), respectively. The output of the backward and the hidden layer at time t is listed below [46]:  
\[\overrightarrow{h}_{t}=L\bigg{(}W_{t},\ \overrightarrow{h}_{t-1},\ c_{t-1} \bigg{)} \tag{10}\]  
Figure 3: Illustration of an LSTM model (**a**) and a Bi-LSTM model (**b**).  
\[\overset{\leftarrow}{h}_{t}=L\bigg{(}W_{t},\overset{\leftarrow}{h}_{t+1},\epsilon _{t+1}\bigg{)} \tag{11}\]  
where \(L(.)\) indicates the hidden layer process of the LSTM hidden layer. The forward \(\overset{\rightarrow}{h}_{t}\) and backward output vector \(\overset{\leftarrow}{h}_{t}\) are \(\in\ R^{1\times H}\), and they must be combined to obtain the text feature, where \(H\) indicates the number of hidden layer cells:  
\[H_{t}=\overset{\rightarrow}{h}_{t}\searrow\overset{\leftarrow}{h}_{t} \tag{12}\]