# Domain Knowledge Graph Question Answering Based on Semantic Analysis and Data Augmentation
## 5 Experiment and Analysis
### Question Intention Classification Experiment
#### 5.2.3 Experimental Results and Analysis  
(1) Model Comparison Experiment  
To evaluate the performance of the RBMA model in text classification tasks, we conducted comparative experiments between the RBMA model and seven other models on the _augmented_train_set_ and _validation_set_. The accuracy, precision, recall, and F1 scores were calculated for each model. The experimental results are presented in Table 6. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **Precision** & **Recall** & **F1-Score** & **Accuracy** \\ \hline DPCNN & 0.9456 & 0.761 & 0.8236 & 0.73 \\ TextRCNN & 0.951 & 0.8218 & 0.8817 & 0.78 \\ BERT & 0.978 & 0.8762 & 0.9215 & 0.85 \\ BERT-BiLSTM-MultiHeadAttention & 0.9846 & 0.9374 & 0.9591 & 0.9 \\ Roberta & 0.9805 & 0.9004 & 0.9371 & 0.87 \\ RoBERTa-BiLSTM & 0.9836 & 0.9459 & 0.9628 & 0.91 \\ RoBERTa-MultiHeadAttention & 0.9856 & 0.9374 & 0.9595 & 0.89 \\ RoBERTa-BiLSTM-MultiHeadAttention & 0.8999 & 0.8965 & 0.8969 & 0.87 \\ RoBERTa-BiLSTM-MultiHeadAttention & & & & \\ (trained on augmented data) & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Model comparison experimental results. Figure 9: Statistics of each dataset. Table 6 shows that the RBMA model outperforms the four baseline models, namely DPCNN, TextRCNN, BERT, and RoBERTa. It can be seen that RBMA compares with RoBERTa, having the most significant improvement in precision, recall, and F1 score by 1.44%, 6.11%, and 4.03%, respectively. This improvement results in a 6% increase in the accuracy of the final answer. The results indicate that the RBMA model performs better in intention classification tasks than the current mainstream text classification models. During the ablation experiment, the RBMA model demonstrated improvements in various evaluation metrics compared to the RoBERTa-BiLSTM and RoBERTa-MultiHeadAttention models. This validates the effectiveness of combining the ability of the BiLSTM layer to extract contextual information from sentences with the multi-level feature representation of the MHA mechanism. (2) Impact of Text Data Augmentation on the Model  
There are seven categories of intention for household-registration-related questions. The specific definitions for each category are presented in Table 2. In order to explore the RBMA model's ability to classify each intention category and the impact of text data augmentation on the model's prediction performance, this study conducted comparative experiments on the _unaugmented_train_set_ and _augmented_train_set_. The model's prediction performance was then validated on the _validation_set_, and the evaluation metrics for each category were recorded. The specific data statistics are presented in Table 7. The average values of each evaluation metric and the loss function curve comparison are presented in Figures 10 and 11, respectively. According to the comparison of the evaluation metrics for each category in Table 7, text data augmentation significantly improves the prediction performance of the model for each category, which confirms the significant improvement in the model's training performance through text data augmentation. However, we found that regardless of whether the model was trained on the _unaugmented_train_set_ or the _augmented_train_set_, the recognition accuracy for class 2 was lower compared to other categories. Upon analyzing the class 2 questions in the dataset, we found that questions related to processing procedures were highly colloquial and some of them were easily confused with class 3 and class 4 questions. For example, 'What are the procedural requirements for applying for death registration of a household?' and 'What are the procedures and documents required for changing the