## 2 Related Work
### Bi-Lstm  
BI-LSTM is a method of adding a reverse LSTM layer to the existing forward LSTM layer, so that the final hidden state outputs a vector connecting the hidden state between the two LSTM layers. This reverse vector captures the hidden properties and patterns of data that have been ignored in general LSTMs. This allows the surrounding information for each hidden state to be balanced in both directions and solves the problem that the existing LSTM experiences a bottle-neck problem in which information is lost. Bi-LSTM uses two LSTM layers and is implemented by adjusting the data order of the layers. The first layer analyzes the data pattern in the same forward direction as before, while the second layer analyzes the data pattern in the reverse direction. The overall results are analyzed by linking, adding, or averaging the results analyzed in the two directions. It is necessary to first look at the equation of Recurrent Neural Network (RNN) to represent Bi-LSTM mathematically [30]:  
\[h_{t}=\tanh(W_{xh}x_{t}+W_{hh}h_{t-1}+b_{h}) \tag{1}\]  
\[y_{t}=W_{hy}h_{t}+b_{y} \tag{2}\]  
where \(h_{t}\) refers to the hidden state output in the RNN neural network [30]. \(W\) represents the weight connecting input \(x_{t}\) to hidden layer \(h_{t}\). The hidden layer \(h_{t}\) is calculated by multiplying the input value x and the previous hidden layer \(h_{t-1}\) by each weight and adding the bias vector \(b\). Output layer \(y_{t}\) is calculated by multiplying the hidden layer by the weight \(W_{hy}\) and adding bias \(b_{y}\). However, RNN has a disadvantage that when too much information is accumulated, it is difficult to continue information due to the long dependency period. To compensate for this, a new method has begun to be used. LSTM is a type of RNN, which is explicitly designed to avoid a long dependency period, allowing the model to remember information over a long period of time on its own. Although it is a chain structure such as RNN, each iteration module has a different structure [31]. Instead of a simple natural network layer, four layers are supposed to exchange information with each other in a special way. \[f_{t}=\sigma\Big{(}W_{f}\cdot[h_{t-1},x_{t}]+b_{f}\Big{)} \tag{3}\]  
LSTM analyzes information through four main steps. As in the Figure 3, the first step is the forget gate layer, which determines which information to discard from the cell state and is determined by the sigmoid layer. At this time, the hidden state of the previous step \(h_{t-1}\) and the input value \(x_{t}\) are received, and a value between 0 and 1 is sent to \(C_{t-1}\) (if the value is 1, all information is preserved). Forget \(f_{t}\) is expressed by multiplying \(h_{t-1}\) and \(x_{t}\) by their respective weights and multiplying the sum added by the bias by the sigmoid function. \[i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i}) \tag{4}\]  
\[\tilde{C}_{t}=\tanh(W_{C}\cdot[h_{t-1},x_{t}]+b_{C}) \tag{5}\]  
As in the Figure 4, the next step is an input gate layer, which determines which information to store among the new incoming information. First, the sigmoid layer, called the input gate layer, decides which value to update and prepares the tanh layer to add to the cell state by creating a new candidate value, \(\tilde{C}_{t}\) vector. These two pieces of information will be combined to create a material to update the state. \[C_{t}=f_{t}*C_{t-1}+i_{t}*\tilde{C}_{t} \tag{6}\]Next, As in the Figure 5, the cell state update is a step of updating the past state \(C_{t-1}\) to \(C_{t}\) by applying the previous two steps. By multiplying the previous cell state by \(f_{t}\), the information to be erased in step 1 is forgotten, and the information to be updated is added by adding \(i_{t}*\tilde{C}_{t}\). In other words, cell state \(C_{t}\) processes deletion information by multiplying forget gate \(f_{t}\) by the previous cell state \(C_{t-1}\) and multiplying input \(i_{t}\) by the previously calculated \(\tilde{C}_{t}\) to process additional information.