## 4 Performance Evaluation and Discussion  
### Performance Evaluation  
Comparing the experimental models together, there was no increase in validation F1-score or significant change in validation loss even when the learning parameters were simply increased, but there was a significant improvement in performance when the attention layer was added. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**F1-Score** & **Loss** & **Validation F1-Score** & **Validation Loss** \\ \hline
0.9401 & 0.2250 & 0.9219 & 0.3433 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Final model training results. Figure 10: Final model training results. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **F1-Score** & **Loss** & **Validation F1-Score** & **Validation Loss** \\ \hline Dropout & 0.9571 & 0.1692 & 0.9076 & 0.4069 \\ Stacked Recurrent & 0.9489 & 0.1915 & 0.9061 & 0.4053 \\ Bidirectional Recurrent & 0.9576 & 0.1647 & 0.9090 & 0.4031 \\ Bidirectional Attention & 0.9512 & 0.1878 & 0.9142 & 0.4072 \\ Optimized Bidirectional Attention & 0.9401 & 0.2250 & 0.9219 & 0.3433 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Comparison of the performance of each model. Finally, the LSTM model showed the best results after adding the attention layer to the dropout and bidirectional recursion layers and optimizing them, with a validation F1-score of 92.19% and a validation loss of 0.3433, the results are shown in Figure 12. Compared to the basic LSTM model, both the validation F1-score and the validation loss improved, indicating that the model was improved. A comparison of this model with other methods is shown in Table 8.  
The confusion matrix of the best generated LSTM model is shown in Figure 13. The detection of \"input error\" has the highest accuracy rate of 94%, followed by \"merchant error for PG or financial company\", \"other\", and \"franchise store error for PG company\". The detection of \"PG company error\" has the lowest accuracy rate. This is almost of the same order as the distribution of labelled data observed earlier in Figure 8. As the amount of data in the dataset increases, so does the accuracy of the resulting results. This also demonstrates the usability of the model, which will be more accurate in future detection tasks after training with a larger number of datasets. The high accuracy of 92% for the F1-score is due to the use of price information provided by the financial companies in the dataset. This is information that has a conservative bias compared to other industries and is therefore largely stereotyped, refined, and has the correct sentence structure, allowing for better accuracy than is possible with general linguistic questions. We can therefore expect similar accuracy for information about other  
Figure 12: F1-score and loss for each epoch of each model. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Simple RNN** & **LSTM** & **GRU** & **Improved LSTM** \\ \hline F1-score & 0.9663 & 0.9603 & 0.9138 & 0.9401 \\ Loss & 0.1300 & 0.1531 & 0.3012 & 0.2250 \\ Validation F1-score & 0.9057 & 0.9076 & 0.8871 & 0.9219 \\ Validation loss & 0.4474 & 0.4187 & 0.4436 & 0.3433 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Comparison results by model. financial institutions not used in this study, and if the model were applied to payment systems for e-commerce, we could set up real-time monitoring systems to determine which payment actions are failing based on response values. In practice, however, an error rate of 8% would have a negative impact on many users. Since most of the data used for learning is automatically tagged data, this error rate could be the result of incorrect data being included in the learning data due to a lack of validation. If humans were directly involved in the data labelling process, constructing the correct learning data and refining the method, a higher level of accuracy could be achieved.