## 3 Design and Implementation
### Implementation of the Model
#### 3.3.4 LSTM Model with the Dropout and Bidirectional Recurrent Layers  
The previous experiments confirmed that simply adding layers had little effect on performance improvement, so the dropout and bidirectional recurrent layers [15], which were considered effective in improving natural language processing performance, were added to the basic LSTM model. As shown in Figure 8, after four epochs, the F1-score and the loss of learned data gradually improved, but again the validation F1-score did not change, and the validation loss gradually increased and overfitted, the results are shown in Figure 7. After 10 epochs, the validation F1-score was 90.9% and the validation loss was 0.4031, the results are shown in Table 4. Although the bidirectional recurrent layer was applied, it did not significantly improve the performance.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**F1-Score** & **Loss** & **Validation F1-Score** & **Validation Loss** \\ \hline
0.9576 & 0.1647 & 0.9090 & 0.4031 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Training results of the LSTM model with the dropout and bidirectional recurrent layers.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**F1-Score** & **Loss** & **Validation F1-Score** & **Validation Loss** \\ \hline
0.9571 & 0.1692 & 0.9076 & 0.4069 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Training results of the LSTM model with a dropout layer.  
Figure 7: Training results of the LSTM model with the dropout and stacking recurrent layers.  
3.3.5. LSTM Model with Dropout, Bidirectional Recurrent and Attention Layers  
To solve the gradient disappearance problem on the structure as the input sequence becomes longer, the dropout, bidirectional, and attention layers [16], which refer to the input sequence at each output time, were added to the basic LSTM model. As shown in Figure 9, the validation F1-score gradually increased after five epochs. After 10 epochs, the validation F1-score is 91.42% and the validation loss is 0.4072, confirming the performance improvement. The results are shown in Table 5.  
To avoid the overfitting caused by this, batch normalization was applied after adding the dropout layer. As shown in Figure 10, after 10 epochs, the validation F1-score is 92.19% and the validation loss is 0.3433, as shown in Table 6, the performance has improved.  
Figure 8: Training results of the LSTM model with the dropout and bidirectional recurrent layers.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**F1-Score** & **Loss** & **Validation F1-Score** & **Validation Loss** \\ \hline
0.9512 & 0.1878 & 0.9142 & 0.4072 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Training results of the LSTM model with the dropout, bidirectional recurrent and attention layers.  
Figure 9: Training results of the LSTM model with the dropout, bidirectional recurrent and attention layers.  
The performance was compared in the basic LSTM model by adding the dropout, stacked recurrent, bidirectional recurrent and attention layers. As shown in Table 7, the performance is best when the dropout, bidirectional recurrent and attention layers are added. The final model structure is shown in Figure 11.